{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ef03480-a7f5-401f-bd7f-563a2ed4eb2f",
   "metadata": {},
   "source": [
    "# 语言模型的目标\n",
    "\n",
    "语言模型的目标是建模自然语言的概率分布"
   ]
  },
  {
   "attachments": {
    "ad44b9b3-fffd-465e-813e-1fa785bafb31.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAARMAAAD3CAYAAAA+C7CYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAOxAAADsQBlSsOGwAAIABJREFUeJzs3Xd4VMXewPHv9t30HhISIJCQ0Htv0gVEqnSlWFERvPjq\nFcXeALsIoqKCgIB4EREbIFVCkxJaQhJSgPTetu++f6wEIiGNTTbAfJ6Hh7Nn58yZs8n+MmdmzozE\narVaEQRBuElSRxdAEITbgwgmgiDYhQgmgiDYhdzRBRBuHzExMUydOpXBgwdTUFBAeno6qampbNmy\nheeee46PPvqII0eOcPbsWZ544gkmTpyIVCqlWbNmBAUFERMTQ1BQEOnp6cyZM4f169ezYcMGunXr\nRqNGjbh06RIPPfQQ7dq1c/SlCuUQNRPBbsLCwggJCSEiIqL09ciRI9m1axcAcXFxXLx4kYyMDGJj\nYzEYDGRnZxMUFMSePXvIyMggJyeHgIAAgoKC0Gg06PV6srOzadu2LQUFBYSFhTnyEoUKiJqJYDdR\nUVGkp6fzxx9/YDabSU9Pp1u3brRo0QI3Nzd27NhBfHw8aWlpbNmyhYiICFatWoWTkxMymQy5XM6J\nEydQKBT069ePU6dO4efnh4+PD3v37sXLywuDwYCTk5OjL1Uoh+yVV155xdGFEG4PXl5ehIeHk5OT\nQ3Z2NhaLhQYNGhAeHs7q1at56623cHZ2xtXVlXnz5rF582bCwsKYPn06Go2G7t27k5WVxaBBg+jQ\noQMNGjRAJpORm5tLREQEvXv3Jjc3l+DgYEdfqlAOUTMR7CYmJobPP/8crVaL0WhEIpGg0WiwWq3k\n5uZelz4nJ4euXbuSkZHBt99+i7OzM926dSM/P5+lS5fi7OxMVlYWycnJNG7cmAMHDtCnTx8HXJlQ\nFRIxaE2wp7i4OPbs2cPx48fR6/XodDqWLVvG4sWLycjIwGQyYbFYUCqVHDt2jGnTpjFo0CAMBgNf\nffUV77//PrNnz2bmzJn06tWLs2fP8uuvvzJ//nxHX5pQCRFMBLuyWq1YrVYkEgkSicRuedorL6H2\niGAiCIJdiK5hwWEsFgs5OTmOLoZgJyKYCA6zYcMG3n//fUcXQ7ATEUwEh7BYLJw8eZKCggJRO7lN\niDYTwSGys7NxdXUlPz8fk8lEQECAo4sk3CQRTASHyszMxNfX19HFEOxA3OYIgmAXYgSsUC3FuiJM\nZqPdxn0UafNRlSjskheAm5OH3fISqkcEE6HKYi+fYcFXM3F1csdisTi6ONcp1hXy5L0v06/dCEcX\n5Y4kgolQZQu+msnMEfNo06yzo4tSrrzCbF77ei6dw/virHZ1dHHuOKLNRKgybzc/Gvo2rlLaQ38d\n4dXn37puf15ePo8+MIdLyZfJy8u3a/k8XL1RKlRIpTK75itUjQgmgt2ZzWaWvr+C+QueAiAjPZOz\np6IB2LRuM02aNiY2Jo7XX3iHi0mXOH70JAaDgZPHT2E0GtHp9Bz66wipl9PIy80jKzOb7CwxFqW+\nE7c5gt3JZDJCmjVmyRsfMnbivaz9ZiOhzZvy195ItCVaJEgoLi5Br9Mz95H/Y+Ddd7H49Q8YOPQu\n9u06QJduHbkQl0BgUADvvPoeBr2BV9550dGXJVRC1EyEWvHa4oWMGH03kfsOM3XGBB6cPZ2o46dp\nHhFGu45taN22Jc3CQujaozOTH7iP0OZNmTZzEkkJycTFXiAxIZngxkFoS3QEBgXi5i7aQOo7MWhN\nqLJHPxzBk+NfxMPVu9K0+XkFWK0WPDw9MBqNGA1GnJydMBqNACgUCvLzClCplKjUKnRaHRonDdoS\nLWqNmqLCYhQKOQqlArPZDIBSqaz0vP/5eBqrnv0TJ5XLzV2sUG3iNkeoskCfJiz731uMvWs6FmvV\nuoZTrm1jzaydcl0RnRQFgFKurt0TCeUSNROhygwmPfOW3YeXmz8qRf37wsqkMh4fuRAPl8prToL9\niWAiOExqaioJCQn07NnT0UUR7EA0wAoO88EHH7BkyRJHF0OwE1EzERwiOzubo0ePAtC4cePShbuE\nW5cIJoJDWCwWpFIpmZmZeHt7I5WKSvKtTvwEBYe4NniIQHJ7ED9FQRDsQowzEaol/lIOBUV6u81n\nkpeXh0eO2S55yWVSWof62SUvofpEMBGq7Oe9Mby4dBctm/lhsdirqc0K2CcwRSdk8sDIdvzn/h52\nyU+oHtEAK1RZ+wmf8c7Tw2gWXD8HhekNJqb9dz17v56Jm7PK0cW544g2E6HK/L1dcK3ml3Trj9+X\nbq/49D17F6kMlVKOWilHLhO/1o4gPnWhVu3a8QtXKr8Dh4zAoNdz6WISp6OOYzAYuHwpmeSkBADS\nUlM4dvQgFouFSxeTyM3NdmTRhWoSbSZCnVn+yRL+8+zLzJ/zIKPHT+aXrT9gNpvRlpQw4+EnOBS5\njzNRJ0iIjyUp8QJTHnjY0UUWqkHUTIRa98Hi11jzzeelr7v16MPkaQ+Snp5KZmY6Xt4+gITft/2I\nm7s7F5MTAWgQEOiYAgs1IhpghSob8ti3vPrEEHw8nat8TElJMUaDAZlMhkwuR63WoNfpUGs0lJQU\nI5PKMFvMODk5U1xchEqlxmwyYbVaUWs01S7jff9Zw/5vZuHiVPncJ4J9idscoco6tghg9uubmTm6\nM5Z6+Dfo4MkkAJw19luHR6g6UTMRqsxisfLOV/sxW6yoFPVvBnipVMITE7ugUYtg4ggimAgOc/78\neWJiYhg5cqSjiyLYgWiAFRxm6dKlrFixwtHFEOxE1EwEh8jIyODy5cu2hla1mpYtWzq6SMJNEsFE\ncKjMzEx8fX0dXQzBDkRvjlCpgoICTp06RWpqKlqt1u55u7m52TXPqggODqZ169b4+PjU+blvV6Jm\nIlTqzTffJDw8nKCgINRqNfb8lcnNzcXT09Nu+VVVcnIyUVFRdO3alaFDh9b5+W9HIpgIFYqMjKRR\no0a1VnvIyspyWO1Ar9ezatUqQkNDGTVqlEPKcDsRvTnCDaWmprJjxw6H3IbUBZVKxaxZs7hw4QLn\nzp1zdHFueaLNRLihI0eO0L1799LXJl02cdvHYdLZb2k+q9VKmj1mbbNa0BddpuWYw2g8qz7TvVwu\np1evXhw+fJgWLVrcfDnuYCKYCDeUnJzM3XffXfr65LoQGoT0pHGXR7FWcXnQupSfGcfZzV1pf38q\nMkXVnx8KCQlh+/btpTPmCzUjPjnhhrRaLZprHrZTOgfQuOUINC6+OLn6l/57YuGvpdtxF438uvcy\n//sjASdX/zLbTq7+fPzNcXYfzuDBZ3/ifJKhdP/Q+1fzn9e34+Tqz0PPbcXJ1Z+TMUWonX2RKb3I\nyJNxMd3KxXQr8ZdMGCyumCXuZcoR0LQXUpkasLJx40ZycnKqdJ1qtRq9Xm/XhuU7kaiZCDfl0LEE\nTCYLefklrN9yhLx8LadjLnN3/1Z8tnoP+w/HcXf/VqXpAxu489hza1n84jg2/3qc9q2DAQgO9CSw\ngQc799naLgqLdMQnZbJqYyTPPzWMybO/5O0FY5j30kaemT2YiY9+Tse2jViz9MHryrRixQpOnjpP\nSkrKDcs9b948O38SgqiZCDdl1cZIMnMKWbf5MHsiz/PM7MGMGdYBgD2R50u3AQwGE1t+O0nSkbc5\nceYix04ll8nrv0/ezSdf7UKrM3LkRCLTxnUnN78EgN5dQ7mrZzgtmwcwbVx32rUKusGk1lby8vLQ\narVkZ2eX++/IkSO19nncyUTNRKgxrdZASnoeW1c9yb3TP2XiqM489eIGCot0DO3fkqF3teLHX08w\ntL9tqLxCIUOhkLFw0RZy8oqR/WuuVqVSzvNzhjHygaV8t+whFi7aQpf2TUrfA9uTwcB1x14l4bnn\nnmPNuh8YOXIkXl5e16XYtm2bfT4AoQwxzkS4oSVLljBlypTSruFTGyLoeNcc1M62L6jFYkGnM+Hk\npKSoWIeLs5rCIh0atQKLxYpSKcdkMpduX5GbV4ynhzNGoxnFP1MZFJfocXayTVZdUKjFzVVDfoEW\ndzcNVqsVnc6IRqMsTVdSYsCKtfSYK/b+8BRtpyQgU7jc8Lq2bdvGiBEjrrvWl19+GZms/k2tcKsQ\nNROhyuQqD1IT9hPcfDBWbL05CjkYDUZUCjAailErwWoxIwGMBj1AmW0AFydbWgCjwbZPKQejwQSA\nRmV730l9NZ1cZjvPlXSKf35zrxwDUJCdgMV89TxC3RLBRKiyZoN/4NSGFiRH/4FUVv/WpZHKNUTc\n82eFtRKh9ohgIlSZ0rkhnWYV2DVP8dTw7UMEE6HKDp9OZf4Hf9KhXYi9VvT8Z3XQ2JvORiqR8Pfx\nCzw5qRMTB1d9BKxgPyKYCFU2/4M/uffe7jRp4u/oopSrQ6fmfLV2J/f2DUWjEr/adU2MMxGqzNvD\nCS8vV0cX44acnFSYTBYxktVBRPgWasWWr96l590TOLj9f4ycPg+T0cCPK5dgtVjwbhBEQW4WcoWS\n5u26YTIaAdi37TsUShWjZs5n66oPuWv0A6QnxwPQse8wFCq1Iy9JqIQIJoLdGfRazp88hJOLG7qS\nIr77+GV6DBlLZkoyTi5uXI7cibd/QywWC7+fO4Gbpw8KlRqNkwv+wU05eWA7RoOOQ9v/R9dBYzi2\n5xciOvbCXQSTek3c5gh2F/n7D8gVClw8vLlv9osU5Wfj4dMA38BGmIwGigpyMeh1+AQE06xVR5zd\nPPH0DUChVGPU63Hx8MbVwxtdSTH+QSG4eXo7+pKEKhDBRLC7iI69aBLRnrY9BpKScJ6QiPZ4+Phj\n0GkJDAmnU78RBIaEYzToadN9IEa9jrSkOPJzMog+cQDfwEZYreDmKeZnvZWI2xzB7vyDQnBx82DX\n/1ahLSlEqVIT+fsmFCo1aqerA8qK8rLx9G2Af3BTigvz8PZvSEjLDpw9so/hU5/g1KFd5OdkoHFx\nR64UawfXdyKYCLVi8IRHqpx2wNgZZV4HNAoFoNewCUgkktLXQv0mgolgd7qSYkoK89j85WJMRgMT\nn3wFpVrDni3f0r7PUM4c3kP8mb9p3q47GZcScHK5OsesXqdl/GML+O7jl8m4dIEuA+5FqdLQuf89\nDrwioSpEMBHsLu1iPFEHduAb2BgPH3+2f/8Ffg2bcDkhhhN//UGbbgMw6LScPxGJVCqj17AJZKYk\n4eXfkD82rODs0b04ubiSn5OJxWIhNuqwCCa3ANEAK9idu5cvly6cQ+3kQnbaJaRSGT4BwTi7etBt\n0BgUKjWevgE0Dm+Lp18gX7/9NEUFufz09ft4+TXE0zeAC2eP0fmuEfy69lMCQ5pj0Nt38S/B/kQw\nEepEqy79cPfxw9u/IbqSQkxGAxdjzyCVSbFYLLi4e1FSlI9EKiEvOwOrFeLPHKP7kLFEHdhBfrb9\nZsQXaocIJkKtUChUqJ2cCWgShsViBiApJorzJw8SGBKOtriApq06cuHMMZ5avJroYwcY+/BzpCXH\nE9AolJ53jye8fQ/ueWAugSHh+AY2cvAVCZURbSZCtUiqsMaNp28AD7+0tPR198FjAZj92uel+7r0\nHwnAwHGzAJj81KsAhLXtVuYYgHGPPl/l8kmlEsSjOY4haiZClXm6qkhKynB0MW4oMzMfnd5UOk+s\nULdEzUSospce6cUDC38mNiYZpaJ+/epIpRJiL6Tz1pN9xfQDDiI+daHKQgLd2fbReKITc7BSv+4l\nrFYrQX7dCPKvv1Mk3O5EMBGqxc1FRdfWAXbJKzExkfj4eAYOHGiX/ATHEsFEcJjly5eTkJAggslt\nQgQTwSGysrIYO9bWYxMdHU1EhJi39VYngongED4+Pvj4+JCVlYWPj5hq4HYggkk9kpOTwy+//OKQ\nc0+dOrVKY0jsbe7cuahUdbsGT0pKynUr+gk3TwSTembatGkOOa+jJmEeOHAgrq6iB+Z2IIKJUC2G\n4hQsxiK7rZszbEBrPD097JKXRCJH5dbULnkJ1SeCiVBl+Rf/IG77eDRuQfaryVit5Njp9kpXeJmg\nLq/j32auXfITqkcEE6HK4raPp33/+Xj4Nnd0UcplMmrZv3ke3s0fQK7ydHRx7jgimFwjJyeHqKgo\nLl++jFZb9/NnFBcX4+zsbNc8VSoVfn5+tGnThsDAwJvKS+kcgMrJ67r9x08l06HN1ad6I4/Gk1eg\nZdiA1mW2AVLS8vhs9V76dAtlcL+Wpce8u/wPHpzcC08PZ77bfJjJY7pWu3xyhQapTI1EqgBAp9Oh\nVovlMeqKCCb/2L17NwcOHKBVq1ZERESg0WjqvFEyPz8fd3d3u+ap1+vJyMjg66+/5oUXXrBr3ld8\nvPJPvv5wBgBRZy+Rkp6PVmcAKLMNsPr7g3TvFMJX6//C19uV9q2DAdjy+0niEjL4bPE0tm6PYvKY\nrkQejadbxxCMRjOpGfno9SYADEYzwYGeyGRSXF3KDxZLlizhiSeeqJXrFcp3xweTixcv8s033zB7\n9mw6derk0LLk5OTg5XX9X/6bFRYWRq9evVi+fDndunWjY8eOdsv78f+uIzUjn2Xf7GbfoVi+/WQW\nsQm2J4snz/6C8fdc/UwNBhNHTiTy3JND2XswloWLt7B19ZMABAd68vEbkxg9cxkSiYQ/90czoHcE\nEx/9nMULx/Hp17tZ8tJ4pjz+JeuWPcTk2V8gkUhYt+yh68p07O9jZGdnc+LEiXLLfPDgQf7zn//Y\n7TMQbO74YHL48GH69OlT52MdHKF3797s37+fDh062GVMiVZrICU9j62rnuTe6Z8ycVRnnnpxA4VF\nOob2b8nQu1rx468nGNrfdjujUMhQKGQsXLSFnLxiZLKyM2AolXKenzOMkQ8s5btlD7Fw0Ra6tG9S\n+h5QOr3Av4+9VsdOHVGo3WjUqFG5wTk3N/emr1243h0fTJKSkujcuXONjtXlx3Hhz6lYzTq7lMVq\ntZJqp54Ni1mHoTiVdlMSkKttK+I1btyYH374AbPZjFxegx+91YJEcvVLrFLJWffpQygUMr7/4hFc\nnNWMG9ERjVqBxWJFqZQzbVw3LBbb7aJEImH9Zw+Tm1eMp4czRqO5NK8v3r0fgG4dQ4iLfAO1WsEz\ns4fg7ma73Xxx7nAAViy2jcP5fPH95T65bDHrwGqhffv21b8+4abc8cGkpKSkRo2eFpOWMz90JKzD\nJDz8wmuhZDdHIpGSdO5X4n4fRcSo/f/sk6DRaCgpKcHNza2SHK6n8W7HwZ8X0KLbTKxWS+n+gn/+\nL/pnmtbiKuSVlnf9vsJrtkuybf9rc67uyy8n3bXbWZdPAiBVuCDUvTs+mMDNjP6U0DCs/3V7M7IK\n8PO5+mX96ru/GNA7nCbBtmdQEi9mlW6fOH2R73/+m8mju9A6oiFga1tY9OnvvDhvOBKJpMa9G03b\njuXojiU1ubByNRu4ntg/xpKaloZUrrFLnka9AYXKPqv1STQRtJu6pUztSag7IphUU05ODrt27WLM\nqBFIpNd/fLl5xaz+/iDPzB6C1Wrl6MkkLqXmotOZSl+7Ol/tgfj0m93MmNCD1d8fpG/3MO4Z3BaD\n0cTna/bi5eHEEzP7l/ZupGXk08DPneycIgxGM/mFWtQqOemZhXRoHYxEIkGhkJXmfW3twR4kUjnN\n7/7JrnlmZmbi6+tr1zwFxxDB5F/MZjPLly+vMM25c+fIykyhSzm9kt/9eIQ9ked5ZvYQ5i7cQKe2\njflt1xnuG9mp9HXXDk1K0/t4uTDliZX8vm4uCxdv4Z7BbQEYPrANB45eYNRQ273/kROJfL5mHy2b\nB9A4yIuNP/3Ng5N78er7P3PPoLa88M6PDOwdwfNPDbPbZ/FvFqOOpGUPUhL/N1aLufIDqijdDnlY\nTbbu5xbvHkfuYv8eMaFyIpj8i9Fo5NVXXyUzs/x1WnJzc9m8eTMzHpjCiTWvlHnv1LnLXEjK4v1X\n7mP2c2txc1UTl5CBSilHggSNWkFcQgbdOoQAkJNbzKlzl5kypisfr/wTk7lsTWLtpw+y4tu9ZOUU\n4eKsorhET9NGPpgtFibc24nmzfzp16M5D0/tTWpGPvFJtbu2TPSzXWj99NeoPBvU6nluxtEXehGx\n6AgytWg3qWsimFSTp6cns2bNwmLWX9e9Gt7Mn7eeH41SKWfJS+NwcVZTWKTjhbnDUSplLHpxHIVF\nOpydbG0EXp7ObF39BAWFOtzdNBSX6AFwcVbzwSsTAHhkWh+mju2Ki7OaVR/NRKGQlfaCyOVSXpw7\nHLVawdvPj7murBIAq/1qEFK1Cze7jsSpc+cp0Wrp1rEdAOmZWfh6e3HkxCnatghHo7nJEatWK3Z7\nClGoFhFMashqMWIxG8hJP4ebl62mcWWFBZPRhFoJJqMOjQrAjMVsxmIGjQosZgPX3iU4aySYjDpU\nCtsxAMprtq/kJQFMRmPpV8VsAoUczCY9SsWVc9uOkUgkpCcdwqi7pjvEwZZ/8x27/jpI64gwPljx\nDcGBDZg05h6ef/M9XJ2daf5/TW4+mAgOI4JJDckULoQP/52oX4YikamQ1Le/hhIpcpUnrSecdnRJ\nAEjLyCTA35eQRkFkZufQtUNbnn50Bg/Pf5HHZ05l09bfUNmpV0dwDBFMboJLgx50mlVQecIqqq3h\n9PVBanomeyOPEJuQRNLFyyjkCmY/+zIjBt3Frzv3cj4+kd1/HUYmk9KpXWt8vMRTv7caEUxuglZv\n4te/LmAwmuySn8FgQKlMs0teaqWce/qGIq9g2HldatOiOYXFxfTv1Y2QRkGcOH2OAb274+/ng8Vi\nYdnXa5k6biSNggJR1GR0ruBw4qdWQyaThSGPbyCksS+e3tUfTXpj9hmafz7mMu+tOcKO5RNRKR3/\nYzYYjQzq05NT52KIibvAgnmP0TioIc+/8R5FJcXs3bKWles2ceTEKV6e/6SjiyvUgON/y25R6347\nS+fOYfTs2bLyxA7Qu3dr9u87zeZdsUwa2sJh5bicmk7DAH9OR8cyqG9PJo4eXvqe1Wrl7Rfnl75+\n58VnOHs+jqycXIIb2mehL6HuiGBSB7JSL7L/l/WMfvD/WLX4/5j+7BJ2b/kWo16LTluCrqQIZzcP\n2vccQl62bQjXoR0/YjYZGDzhEfb+tIYB42ah0jgTf+oIDZu1IKjprbHOzEdfrKJZk0acjj5P7IUk\n+nTvzP3j7+XLtZs4ExNHanoG0yeOYdr4e0m8eJlfdu6hYQN/2rWKoFV4mKOLL1SDCCZ1YMemL/H2\nb8imz97CydWdwzu3kJV6EaNex+XEGHwaBFNckEdhfg6nDu5EV1JMcUEebbr3R1dcSGF+Dps+e5MB\nY2YQGNKcY3t+uWWCSce2rRg3YgixCUns2HuAR++fRNTZaC4kXSS/sBAnJw1nY+I4ez6OM9FxHD1x\nGtqDwWAUweQWUz9a525jupJiDm3fTIvOfTHotdw3+0WCQ1sS2KQ5Rfk56EuKMRkNmIwGPLz98PYP\nwt3bj4597+bY3l+RK9WEtu6CxWympDAf/+Bba/b1Rg0D2fzLdpZ9vY7f/tzHu8tX0qVDW1QqJX27\nd6Fv9y54ebrTpX1bZkwai5eHO/Mfm8WMSWMdXXShmkQwqWVypZKB4x9EJpdjMZswGQ0ENA7DzcsH\nZzcPOvYbTsOmEWiLC/HyC8Rg0KEtKuTs0b04ubpz9vAewtp2pSA3i6DQ+tk+U5Fvv/+RCaOGM++R\n6dwzuD8vzJuN2WzG2UmDwWDA28ujdJ7WzOwcdHo9Umn9/bXctWtX6fb58+fJyak/gwIdTdzm1DK5\nXIF3gyAAOvcfSfL50zRtZZs2MSUplvwc2/M0hblZKFRqTAYDIS3bo1SpSb+YQGibzpw/eZChkx5l\n94+rCWrmuMbUmmjTIpyXF39MTl4e8YkXSc/MwtPDjWED+rLrr0OcPB1Ny/BQfvr9T1au+56P3niR\n33ftY9/Bo7zx/NOOLv51Bg0axNatWxk+fDhr1qxh4MCB9OvXz9HFqhdEMKkDao0z0X//RUFuJmaz\nicTok5QUFdCycx80zrZu5ayUJI7v+512vQZzfN9vAAye8DBxUUdxcnWn++CxhLXtisloJOHccUde\nTrXcf98oXF3KTj5lMpmQy+W0aRGOk0aNl6cH6ZlZDOjdDRdnZ1ydnQkNaeygEl9lsVhYsGABr7/+\nOkuWLOGhhx5i/PjxvP766/Tt29fRxat3RDCpAx363F2t9I2btyndbtdrUOm2t7+thuMfFGKfglWT\nRK7AYjJUnvAa/w4kQOmUkUGBV58+9ve9uni5t5cH3l7VX+XPajZhMWiRyGSVJ64CqVTKiBEj6NSp\nE4899hh+fn54eHgwefJkFixYgIeHfVYivF2IYFLLDHotOzd9hdGgRyKV4uHth1Su4NjubUhlMuQK\nJQa9jiYR7WgQ3BRv/yAuXThHx77D2LFpJYEhERzf9ytunr5cij+Li7snU+a+gbu3X51fS/PX9pCy\n8RVK4o8iVdl3fZ+bZ0UVEEarpbFI5fZ5xsdisaDT6YiKiuLQoUPk5dnmmhw9ejQ9evQgIiKCgQMH\n2uVctwMRTGqZUqUh43IiAY3DOHdsPy069sbTLxCNiysRHXtTlJ9DVupFlCoNSCTs+nEVTVt14ovX\n59C8XTckgJOLGyEt2hHWpgt/7/0Fva7EIdcikUgInPhqmX16vZ5t27Yxdmz1e18yMzNrPKG3yWQi\nMTGR0NDQah9bVVKplMGDBwPQrVs3AN577z0A/P39SUpKQqOxz/SVt4P622x+G1OpNTQJb0fXgfdy\n+UI0RoMeo942jF4qlWEy6DAZ9Di5eiCRSrlw5hi5mWns3rIaZ1d3Th7Y7uAruGrVqlV89NFHNZpH\n95NPPuGDDz6o0XnXrVtX+sWuSy4uVyddcnNzQ6FQ1HkZ6isRTOqAVCpF7eSMXK7AL6gJAGaziV/X\nLiO0TRfyslJp0qIdao0zXQeNwmqFcY+9QFpyPBaLhXGPLcBqMRPSogNdB46iQXDt/TWuDr1ej7e3\nN08++SSHDh2q1rE5OTn06NGDadOmcf78+WodazQayc7OplGjRiQnJ1frWKH2iGBSBxo0akZgSDiN\nw9vSILhvyFtnAAAgAElEQVQZYW274hcUQvP23SnIzeKhFz4hMyUZi8VCqy79GDLxERRKJVarheZt\nu2Ixm7lr1ANYrVYO7/wJpbp+VK3lcjnjxo2jadOmdOnSpVrHuru7M2zYMLp06UKzZs2qdaxOp+Pp\np59m6tSpNVqyQ6gdos2khqyAtIoLZg2e8AgAoa2vtg106D0UgBYdewHQf/QDZY4JDm3FlLmvA+Dh\n4w/A+McWVKuMUqmkVtdLll3TayKrZg/KzRzr6upaui16VOoPUTOpIU9XFTExFx1djAod/TsOD1cx\nDaJQN0TNpIbu7RfGvf3COB2Xic5gn0mbL126hIeHR5lGvprSqOXsWzm1yum/+uqr6/YdPny4dBXA\nypw5c6Za5btiwoQJNTpOqH9EMLlJrUPtt4DU5x++RJMmTViwoHq3M/bw4IMPXrcvJyeHyZMni3YJ\noUpEMKknEhISGDBgAAD5+fm4u7s7uESCUD0imNQTgYGBDBkyBC8vL/R6vaOLIwjVJoJJPaFSqSgu\nLi7dvlN06tSp2r051x57bc9OVSUmJjJixIganVO4MRFMBIdq1qwZzs41e84nPDy8RoE3JMQxD0re\n7kQwERzq008/xdfXfo3YguOIYPIvUqmU9u3blz7gVZdMJlOZmbzqG6M2g7jf78Wote8C6al2ycWK\nUZtFi1H7cPJuZ5ccheoRweRflEol27c75kG6+j4FYNR3oTQMG0CjiBlYrRZHF+c6BdkXOLulD+3v\nT0GmuPmxOnZhMWE16rCa9Pyy+QcupqTgUUlXe3pmFkrvIHx8fCpMl5mZidVqxc+v4ukoUlNTCQio\neOmQ/Px8CgsLCQoKqjBdUVERbm5ujB8//rr3RDARqkyu9iQ4fDAqTdlu6+OnkunQplHp68ij8eQV\naBk2oHWZbYCUtDw+W72XPt1CGdzv6py27y7/gwcn98LTw5nvNh9m8piu1S6f2qkTMUecsD3s4CBW\nC1aTHqtRDyY91mtWqD98/CQNfH1o3aJ5hVmciYnFy9mHNm3aVJjup59+wmAwVDqnyvHjxxkyZEiF\naSIjI0lMTGTYsGEVpouKimL37t0imAg3RypTl1sj+Xjln3z94QwAos5eIiU9H63ONiPbtdsAq78/\nSPdOIXy1/i98vV1p3zoYgC2/nyQuIYPPFk9j6/YoJo/pSuTReLp1DMFoNJOakY9eb1uG1WA0Exzo\niUwmxdWl7OMCtvJJWLt2LcOGDauztZutZiNWox6rvhBu8DyUk0ZDcMMAWoRV/GBjAz9f/IKCaNGi\n4vl+//77b/R6faXp/P39K02TkpJCSkpKpekKCws5duxYue+JYCLclEPHEjCZLOTll7B+yxHy8rWc\njrnM3f1b8dnqPew/HMfd/VuVpg9s4M5jz61l8Yvj2Pzr8dJgEhzoSWADD3buOwdAYZGO+KRMVm2M\n5PmnhjF59pe8vWAM817ayDOzBzPx0c/p2LYRa5aWHblborPy/gcfEx0TX+H0BM8///zNXbjVitWk\nK7cGcqcSD/oJN2XVxkgycwpZt/kweyLP88zswYwZ1gGAPZHnS7cBDAYTW347SdKRtzlx5iLHTpX9\nsv/3ybv55KtdaHVGjpxIZNq47uTm22aV6901lLt6htOyeQDTxnWnXasgLJbyawBXnpS2Wq3l/qvu\n/CnXsuiKsBRlY85PxVKci9VQIgLJP26bmsm5c+dQKpWlc2McPHiQ7t27O7hUtzet1kBKeh5bVz3J\nvdM/ZeKozjz14gYKi3QM7d+SoXe14sdfTzC0v61tRKGQoVDIWLhoCzl5xchkZf+WKZVynp8zjJEP\nLOW7ZQ+xcNEWurRvUvoe2KZVAK479gontYT5/5nL+o1bbnibs23btipfo9VqAZOhtBEVEThu6LYJ\nJl988QVWqxW1Ws3bb7/Ns88+y969e4mKisJqtXLs2DFmzpxp13Ne2DmF3KSf8WpY/TlMy2WF/KpN\nkVIpiURK9qXDBHd7B79Wj9slT7nam0uxf9I44m6sVgtSYMOyKVhMhfy40jYfy+R7yy4UduW1Xmub\njHnVB/eVef/K/q/fG49em0f7Fp5cPLIQrMW8+NTV9WhemnsXem0eK5eMK/3/2uPB1ptjMesACVOn\nVv2JaQCFXA4mHRadAatRW61jBZvbJpgATJs2jddee40TJ06U7nvxxRcBWL9+vV3PZTGVkJv0M33H\nL0MqrXg4eFxCBg383HBxtjUWFhXrSrfLew2g1xtRqWzzixqNZhSK6g85Lym8h+O737NbMAkdvJmo\n9WGkJRxCKrfPPClWqxVJFSeZqoxUpqHFqL+QKao2olZiMSGxGJFajAzr0wlLcW69Xk2wvrutggnY\nJimePn166X2zWq3GarUSGxtLu3Y3P5gpLy8PJycn5FIpEqmi3ECyc985Bva52iq+4aejjBvekYgw\n2zoxl1LySrdffW8rJpMFlUrOs48PRamUU1Ssw6/NM5z682WaNfFl+tyvWbfsoWqXVSqzz5IPVyic\n/Ok0q8CueWZmZtbtCFirFanFhEKfB9f0TNkroN3Jbrtg0qhRI8aPH8+HH34IwDfffFN6m2MP7u7u\nLFmyhKfmPFbu+5FH41n7v8MM7NOCXX/F8MO2YyQkZzF2eIfS10/MvKs0vUqp4NS5RGZO6sW8lzay\n7J0pAAzu24Ln39rMxs9tUz4WFGp5+5PfmDauGzm5xezcH03Txj4cO5WMTmekY9tGtIloSI/O1ZtP\ntbpy9q+n6MwuLMbqLcZ1IyaTkWL5zc/wbrWYUAeG02DcC9cFBonFhNRiRGIx2rbNujKBpK6cPBPN\n2k1beHfZygrTpaRnoJDL+fSDJRWmS8/KxmKx8u2331aY7tKlS/z4448VpsnNzaWgoIAdO3ZUmK6g\noOCGwf+2CSavv/566UNfjz/+OFOm2L6UTk5OAPTp06fKeT377LNs3Ljxhu9brVZ+2PQdyx+/vjfh\nq/UHiIlPI79Ay+dr9vLd8od5+uWNSJCUvo6OTQNs3Z9R5y7Rt3tzki5lk3z56gjYBn7u9OjUlK/X\n/wXA91v/ZmDvCD78YifDBrTC3U1Dvx7N2X84jufnDOOFd37k1LnLtRpMsnZ8QeqmN2g27TW7LXRl\nN1YrKbvXkrL2vzSctgip2WALHmYDEkcOYrtGu1YRTBl7D8MGVrw28WervsPb04P77q14ANnaH35C\nbzDwyPyFFab78MMPmTdvXoVpdu7cSWRkZGmzwI0cPnyYH374odz3bptgcu2TpxKJBE9PzxrnVVRU\nxHPPPXfDKQU///xzZjwwhdTtHcvsLykxcDElh4G9W7B+yxEMBjOFRTqSL9mCxJXXV8QlZNCwgQfT\nJ/Rg4mOf075VcJn8ZkzsyaTHvkBvMKFRK0nLzMfDzTaFYuMgWy+Fr7crbi5qfL1dKS6p3XlQ0ra8\nS8unvsQluH4unu4e3o1jr4yg4YSFyCz2qTkJVSdam27AxcUFb2/v6/5JJBJmzZqFv7//dcckXc7m\nuSfu5oW5w5BJpSyYO4zFn/7OiEFt8PRwKn3t6WGrLXVo04gWYQG8s/Q3Pnh1Av6+tmc2lAo5A3qF\nA/DOC2MY0q8l40Z0IDevhOeeuJvQJn6ENvHDzUVNz87NUKsV9OsRxsDeEbX6mchdvFA419/Z4KWK\nf6YjMJscW5A71G1TM6krV8YtWExarBZTmd6IFmEBtAizPVD10NTeAHRq27j0WH9ftzKvAWZN7lW6\nfeVYpVLOxFG2dWiaBPswe7qtWjznQdu0jj7eVx9iGzHI9vzG2OFla0lX2PsJX0G4ERFMakgq19Bx\nZh7nT7yDRCIHO/QGaEtK0PzTxmMPcpWn3XtfBOFGRDC5CRKJBC+/MejT4sEO4xMsxiKcLPZ6dF6C\nU4MOlSerJW9/vIIH7hvNa+8tZcrYkfj5eLFjXyQASoWCzOxcSkpKeP2/83jzw884cOQYk0aPQCqV\n8sCE0ZyLjSf5UgoKuZwBfXo47DqEqhPB5CbkHfmJiyvn4NGmv926Gu1Vj7AY9SSveJRmz2/FqXFb\nO+VaNXEJSezY8xcSiYTw0Ka888nnLFv0MufjE3jm8Qf5at0PvPJ/c3hp0UfMmreAAH9f3nphPivX\nfk/HtraHAmMvJHLk+CkG9e1Zp2UXak4Ekxqy6Eu4uHIOred/i1ODpo4uTrmyT+7k8upnCVv4W52e\nNzSkMT27dOToiVMUl2hxcXbixOlo9hw4QmADf0q0Wv7YvZ/4pIvMnjGZDT/+woSHnuLh+ycyoHd3\ntFod7q6utAwPRamsZ13QNyE9M4v0zCyycvIqTHfo2ElcnJ3R6SvukYo8egyjyYx69eoK0x05coTV\nlaQ5c+YMcXFxlaaLj48nLS2t3PdEMKkpiRSpUlNvAwmAa5O2mIuW1vl5T5w+x+4Dh3hwyn18v/U3\n5HI5jYMb0rNLBywWCxNHjyAnN49uHdux/9Df5OTlM3PyONxdXZj48FwWv/Qs+w8f4/dde+naoS07\n90XSr2cX+nSz0zNQDmSxWDCbK35Y0Gq1VimdxWJLZ8iqeJlaU3FeFfKyYLVaq5yuPCKY1JQErDfZ\nBVlYVMwT/30VpVJB766d8PPxRiqVIJVKOXEmmoemjMfL8ya6Ymtx0fKKtG/dgkF9e9KmZThr/7eV\nDm1aEnUmGhdnJzIyszl64hRmi4XkS5dpFRHGxFHzePy5VwhvFsJv61fi5elB907tOR19nuISLTMm\njaNNJbOT3Qr8fX1oFR5a6aA1nV5fpUFrCoUcvcHAjEljK0yXX1hU6UOuVwatVZbujhi0dqtJy8hk\n3sK3aNsynLt6dmPF6vVMnzDa9iW7nEpi8iUUipsfZu4I8YnJSCRSOrVtxfQJozl3Pp6xwwdjNJlQ\nKhRMnziGH3/dwdmYOM7GxHEpJZ03nn+azu1sUzsmX07h/15dzIbPP6SouJhFn3yBTCalZfNQB1+Z\nUBExaM1BsnPzaBUexvFTZzl++izNmzXhjQ+Wc+hYFKnpmew7eJQPP1/FGx8sJy4hydHFrZZmTRrh\n4uzE82+8R+LFFAxGI82aNCL5UgoAryz5mDc/XE5gAz9CQxqza/9B/tx/kHumPcrLiz/G3dWVl+Y/\nwcJ3PiTpUgqTxoygedMmjr0ooVIimDhIRGhTvDzdWfifJ9j002+YzWZ+37CSx2dMRq/X07l9awoK\nC3lwynhCQxpXnmE9cyHpIn17dKZT25akZWQReyEJlUrFlLH38PxTj9Gzcweemf0gp6Njkcll/Lkv\nEp1eT+LFy5w9H8eR46fIzS/gyPFTvPD2B+QVFDr6koRKiNscB8nIyiE6Nh6DwcgX779RGjCefult\nnn50Bu1bt+BSShpvfbScd158Bmc7DmarC3KZjAZ+tqdLNWo1zZs1YWCfHhw8dpIdew5w+HgUb330\nGZ7ubvTt3qX0uMupafTo3IEenTvQrVM7WoQ141xsPBr1nbNk6q1KBBMHCfD35ZO3Xrpu/6pPFgG2\nVvOgwAblprkVFBYX88vOPQBcSrV1JX65diMrP3iLPt06U1RczKvPPsUbHyynpKSE5Mup9OjSgcKi\nIgCMRiMRobaeMp1Oj5NG45gLEapMBJM6ciYmloN/n6SouIR7hw7AYrGw/sdtvDBvNlt//xOAkUMH\nsPSrNVitVn7evpt7Bt+Fk0bDg1Nsa5Rs3PILE0YNd+RlVMu1nUkZWdl4ebiz+NMvKS4uIfLoCV5e\n/DE+3p7k5efTonkz4hOSiQhryqoNm9m2YzcL5j7GH3v+4sz5ODF50S1ABJM68tZHnzFp1Aj8vL14\nd9lKnnroAYbc1ZulX60hJu4CcQnJHD15GoCoczGMGjqQNZu2MHH0CPZEHmbb9t2cjo7l6MnTmExm\n3n/tJpdqqGVzH55O25a2J5+jzsbg7KTh6UdnEtjAtvrcq88+BdiWRC0qLuF0TCyNGgbQqGEger2B\ngX164OHuipNGw9OPTHfYddhTdSdH+mzVdxWmuzI50tpNP1WY7lJqOj/9VvGkR7n5BRQUFonJkW4F\nuXkFRJ2LAWzjCGQyKX/uP8iDU8ZzJjqWvQePMmPiGL5Ys5GeXTpQrC1hzIjBqFVK+vXoSr8eXVnw\n5nu89cJ8nnz+NQdfTeWuBJJrt8tr95HL5Xi4u9G7a6fSfSqVkqBA27SWzZtVbT7XW0FtTY40a/L1\nq+td66MvVjP34QcqTPPnvkgi/7bVFitS0TgT0ZtTR5o3bcIj90/kkfsn4uXhjlKpQKvV4unuRgM/\nXzw93Ahs4IeTkwalQsn5+ET8vL1JTL5MiVbLx1+uxs/Xh7z8AgqLih19OYJwHVEzqSN5BYWsWL0B\nALlMTkxcAv17defND5eXpnnzw+XMefB+Vm38kdCQxhQUFfHo9Els+W0nGrWah6dNYPazLzNr8jhH\nXYYg3JAIJjUmqda0A998/E65+/v1vH6B7nn/aiNoEtywdHv54lerfE4kEpDYsfJptSCpZFkPh5PK\nbC2/or22zonbnJqymJEq1OhzUhxdkhsqvhSNMa/8JzxrQu7hT150pN3ys7fiSzFYdEVI7BlAhSoT\nNZMakqqcCBi/kJNv34dPt1EOWTqhIlaLheyj2wh52n6LjwXdv4TY14eQe+4v5M41n7C7VlgtZB/9\nhUaPrgCVMxgKoZ7MSn+nEMHkJnj2vA+lXxP0qbFlpm08ceIE7du3r3Z+ZrPlhmvoVoXVem0xJIQN\nm4u6YXhFh1SLumE4EYuPUnRuP1bT1bk2kpKSaNy4ZkP+zWYzMln1b50KCwtwdXW7ukMiIXToHDTB\nLbECRrVn6Wp9UrMRrGKN4NomgslNcg7tgnPo1eHgVquVl59fxr4nF1U7rzVr1tCsWRN69Kj+NIW/\n//47Wq2W0aNHV/vY6lC4+eLZbUzp67S0NBZ9Mo/165+tdl4HDx4kJi6G6dOnVPvYj15+mREjRtC1\n6/VtTldYpQrMUgVmOWA1IzXbFuJyVINKbn4Bx0+dQy6v+Gt3LjYeNxcXPNzdKkx3+tx5jCYT2/f8\nVWG6mLiEStOcOHWWuIRk/vjjjwrTRUdHk5OTU+57IpjY2ZkzZxgxYgTHjh2jY8fyZ4wvT1xcHH37\n9kWhUJCRkYGfn1+Vj01NTaVt27ao1WoSEhIICQmpSdGrzWg0YrFYmDZtGrGxsYSFhVX52MzMTBo1\nakRAQEC1j71w4QKTJk0iMDCQ3Nzcqq2RJJFhkcsANSaFM0a1bZUBidWC1GIkLSuXdrW8zrCnuxut\nwkMZ3K9XheniE5Px9vSoNF1GVjZ6g6HSdGfPx1eaRiaVYjCZGNA6uMJ0LsZcLkZHlfueaKmys9at\nbXNyVCeQAISGhuLi4kJAQEC1AglAQEAAAQEBuLi41FkgAVAoFAQGBgJUKxgA+Pr6EhgYiLOzc7WP\nbdq0KTKZDHd395tabA3AKpFilqk4FBWDxNUPqYs3SMXf2JoQn5oglJIgkauQuflhtZjAZMBq1GE1\n6hGNuZUTwaQW/PDDD5w+fbraxxkMhpuaQPnrr7+u8bE3Y8GCBSxaVP02IrA9m1NZG0J5dDodR44c\nqdE5q0IilYNSjkRpewTAatJjNeqx6otq7Zy3OhFMakFliz/fSFFRES4uNV83pyZfSnuobEbziuTl\n5eHhUX+XHL1CIlchkatA7WwLKkYdVpPeYfPs1kcimNSCUaNG1ei4nJyc0uVHbyU16Qa/IisrCx8f\nHzuWppZJZEiUTldrLEb9P7UWHVju7DWORTARhJsgUaiQKFSgccNqMWPV5tvG4NSzQYx1QQQTQbAT\niVSGxNnLdutjNpbWWKxmo6OLVifu+GCiUqkwGo2o1eoy+0+fPs1vv9XtSnhFRUWMH1/x3BQ3y2Aw\noFJVbT5VhUKByVS7VffDhw/j7u5eq+f4t+joaEaMGFH62mw2I5VKazQSt1wSCciVSORKJGrX0t3P\nvPKObRGrSmaNe/g/LyCRSCpNN/nRpwEqTffkf1/BWskYmrvGNKPf6KmVpus8uBldh5b/1PodH0y8\nvLzIzMzE1fXqD71169bs2rWLCxcu1GlZDAZDrQaTgoICFApFlYOJl5cXWVlZtdqO89577zlkfaD5\n8+eXbmdlZdVJI7CjGsjrisR6o7X+7hAHDx4kJiaGsWMrXhWtLtR2A+wff/yBUqlk5MiRVUr/999/\nc/LkSe67775aK1N9aIDdunUrfn5+DBo0yKHluNXd8SNgO3XqREFBAf/73/8cXZRad+LECfr1q3jK\nwGu1a9cOrVZbiyVyvO3bt3Pu3Dl69uzp6KLc8u74mgmAVqvlrbfeIjg4mGHDhjls3ENt1UxKSko4\nduwYPXv2rHb7hE6n45tvvmHYsGG1UjZH1UwKCwv5+++/OXPmDHPmzMHZ+faZa9ZRRDD5h8FgYPfu\n3Rw7dgypVIpGo7nhau+1pbi42K6/1BKJBJ1Oh16vp2XLljW+ldu+fTvHjx/HYrGgVqvt+rnc7EC9\nmpBIJJSUlNC2bVv69etXpr1MqDkRTMqRn5/vkOp9lZ+ArQaVSoWbm5tdeiqKiors/rk4omYikUjw\n8vJCWstPCd9pRDCpR27VEbA3IzMz84brsAi3FhGaBUGwCxFMBEGwi9t7FE0dsFqtnD9/nv379xMb\nG0t2djadO3emR48etG7dWtyXC3eMG7aZpOfoWL4lkZxCw527aLTVSrswd2YNK3+y5Pz8fBYtWkRi\nYiL9+/enQ4cOeHt7s3//fnbu3Im3tzcvvfRSlbtjRZtJ5axWK+fOnSMyMpIzZ87g7e1N69at6d27\nN97e3jc8bv+pbE7E5mM033kP4AFYLODjrmDa4EbIZLXzfS43mBTrTIx+4RC9WzegWaAblju0idZs\nsbDnZAqdwt2ZM7Zpmfd0Oh1Tpkxh+PDhzJw587reEqvVyqeffsp///tfEhMTq9RjIYJJxdLS0nj3\n3XfJzMykX79+dOzYkezsbA4fPsyhQ4cYOHAgTz755HV//NbtuMjXvyYztHMQUumd+YdRJpUQeTad\n7AI9W9/ujlpp/8XUyg0mG3ddZn9UHvf1a2b3E95qDEYzH20+xQ+vdUUhv3rLMmTIECZPnszIkSNR\nKBS4ubmVW4P77LPP2LdvHytWrKh0PIUIJjeWnZ3NzJkzmTlzJqNHj77usy4oKGDRokXExcWxdu3a\n0udgtHoz9y44yLxxbXBzqvksdreL9btiGdjJm7F9A+2ed7k39EVaI/6e169Yb09HIndiNps5e+ow\nACXFhZyJOlTm3+WLVx+0i489TXraxTJ5mIxGUi4nUpBf/tT79qBUyLBYrBhMV6vHVquVQ4cO0a9f\nP3x8fHB3d2fdunVIJBLi4uLKHP/YY4/xySefMHXq1Bs+gfvtt9/y+eefI5FISE1NZdiwYfz555+1\ndk31walTp/jiiy9Yu3YtBw4cqDDtkiVLePrpp5HL5axcuZIRI0bw6qtll0l1c3PjzTff5LvvvmPW\nrFml+01mKxqVrFYDSVZmKnExUej1utJ98edPXff7fPb0EfQ6re3356/rl5RIuZxIWkpSrZUTIMDL\nmcKS2nkS3CENsAaDnh/WLyM5MYaUy4mcO32UpqGt+Xnz12RmXMbdwxtXN080Ti706T+SP//YhK9v\nIGaLmbzcLLr2GMR3qz9kyPBJAGRnpTHjkQV1Vv4zZ84QHR1NQEBA6b6pU6fi5OTEsmXLeP/998uk\n9/LyonHjxpw5c4Z27dqVeS86OpqTJ0/y7rvvlu7bsmULH3/8Ma1bt672TPW3giNHjrBu3TrmzJmD\nRqNh/vz5FBQUcPfdd1+XNi8vj9WrV6NUKtm6dSuBgYFotVoeeOABtFotGo2mTHqpVMqjjz5aV5cC\nwKZ1n+Lp5cvJY/sBaNOhJ9+uXIxBryMgsDEX4s4glckICGxMSGgrTCYjRYX5HI7cQffeQ9m0dinB\nTZoTFt6W/Xu28fqSdXVafntxSDDZ8esGGgY3Q6V24oGHnuO3rWswmgyERbSjaVgrYmOiCA1vS0lx\nIb/+9C06bQlBwbZbLoNBj69/Q5qHt2PAkPHk52Vz/OjeOi3/0aNHmTFjxnX77733XlasWFHuMe7u\n7ixfvpzmzZuX2Z+UlMTYsWOJjCy7hm+LFi1ISEi47YJJVlYWy5cvZ+nSpTg52Wq/a9eu5ZNPPiEk\nJITw8LIrEG7evJmCggLi4uJKpyrQaDR8//33fPnllzz00EPXnaNbt27odLrr5qipDVptMdt/Wc97\ny7aSlZlKQvxZXFzcaR7eDmdXdw799Qd3DR5L8T/BIy01mS49BuHh4U1uTiY+vgFIZTImTJuDn38Q\nSYnna73MtcUhwSQvNxOVWoPJZCQ97SJOzq507TGITeuW4uzijtls4kjkDoaPmk7T0FZs3riCqOMH\nkMnkOLu4sWndpzi7uPHD+mWUFBdxKTmOEaOn11n5//0Lf4VMJiv9gvzbgQMH0Ov1172fnZ0N2O75\nryWRSLBYbr+eh8TEREaPHs3582W/NO3atWPVqlXXraGzZs0a+vTpU+6cJ0ePHi03mMjlctLT02nY\nsKF9C1+O6DN/07hpOP6Bjdi6+WuKCvO4b+qTKFVqTp88iFKpKv0/uHEonl7+pKckk5+Xjdli5u9D\nu8lMv8zXn72Bf0AjLiae5+Sx/bTr2LvWy25vDgkmU2bMZ+Wy1+h91z1s3rCC+6Y+CYBKpaFJ0wgA\n8nKzMJtNKJUqLGYzRUX5yGVyJFIp4yY9RmZGCn/t3kZwkzDc3Ot2Ee2KGgz/XfMA23IOycnJfPTR\nRwwfPrzMe6dPn+bw4cNl7vOtVivz58/nmWeesV+h64krS1skJZVtG/Dw8CA6OprU1NQy+w0GA337\n9i03r5KSkhueJyEhoU6CSYfOfTl2eDcajTPakiIah9j+0BgMeho1aV7aUJyeepFmzdtwJuoQCrmC\n4uICzGYTXj7+9Oo3AqlMhtVqpWFwM4wGQ0WnrLccNmitID+HP3//HqlMxh/b1hHRqjN5edmkpyYD\nUG6mUP8AAA9HSURBVFiYT3JiDB6efnTtOZic7AyUKjUGvQ4nZzf+2v0FoeFtaRDYiOQ6rhqGhoaS\nmppaps3kinnz5rFr1y769+8P2J4EbtiwIYGBgQwZMuS69C1btuTAgQM8++yzvPHGGxiNRt5++216\n9OhRulre7SQ4OJijR4/yxBNPlOmRef/991m4cCEdOnQok/7nn38mOjq63LwyMjLKXQ41LS2t3KBe\nm/78YxM+foFkZ6axf9dW0lKSsFjMpdeYm5NJRtol3Nw8CQwKIaCkiNTLiTg5uZCWkoRer2Py9Kf5\n3/rl+PrXfhCsDeV2DX/1SyKpWVb6tr3+y2IvMeeO0zDo6tiN/LxsdLrr/9I0bhJOUWEeMoUCmUxO\nWkoSDYObYTDokEllJCXEENSoGa5utVc7WbT+OBte6YKz+mrsfeutt1iwoPxG30mTJqFSqXB3d+fE\niRN4e3tz1113MXfu3Bue4/vvv+f8+fPI5XI6dep0W8/6tWnTJnbv3l06ZeM333xDQkICr/5/e/ce\nFWW5L3D8y224yFVkVEQFhRBBdpqQB1G2d5flqnboSttesmPHZeHuaJq2RNlKrDBb7pVnYakLLRaE\n5TFR0UrJc1JTRCXoDFiD5OaSiFAMzDgMw8x7/uDIycRLwwsvyvP5x7XkfZ/5/dYafjzv8z6Xv//9\njhnDtbW1LFu2rMPNq06fPs2ECROora1t7y3W19cTEBDQvrq56WYrLyZfYPVc24/juB/tD0UEDAnG\n8n9v61pamvn1lxt3XHfru2wwNDFwUCCXNRcIi4jil/rruLi4oW9qwGRqZkhg1xXC/y76mcH9HVk0\nc4jsbXfYM5EkunxyT2jY7X+B3D3uPkvUu+//P1YMCw4HwNm5bXAtLGJsF0R3Owd7uztOh9RoNGRk\nZLBgwYI7rs/MzKSqqgqj0Uh1dTUrVqxg9+7d9/yMOXPmUF9fj5eX1yO/V2h8fDx2dna88847ODs7\nExQUxIYNGzpceqBWqwkMDCQ3N/e2TaABYmNj2b9/Py+//DKPPdb2SJGXl9dhD7ArhYT+6Xf/40Vf\n3/4dXuvd16/9+zxyVDQAvv0GANDH3bPLYuwO95wBO6hfH6JD/XrtDNhWi5Xc/AoWTB/Mwhl3VnKd\nTsfatWuxt7dn6tSpPP744+3T6U+cOEFDQwMbN25k6NCOp+P/Vn19PVu2bMHJyYmkpKRHvqAAfPrp\np5SUlJCUlHTfa8+cOUNsbCxFRUVERkZ2eE1GRgYZGRkcOnTotjc572Zr0VYaifuTf9sfhl7I3s4O\nbbWO/yr6mc+Tn8TdVf7v113X5phbraQf/Sd1ut67NkflaMeYx7yZPObuA64Wi4Vz585x6tQpysrK\n2hf6xcTEMGHChAcuCrW1tTg6OuLl5UVDQ8M915k8CiRJYs2aNRgMBlJSUh5oq8zvvvuO5ORknnzy\nyfa1UHV1dVy4cIEjR45gb29PUlLSHQPkkiTxj/1XKP2nnl5aS5Ak6Oel4rW/DGOgb9e8MhebI/Ug\nvWk6fV1dHR4eHuh0Osxm8wO/edHr9Rw5coRz585RUlJC3759CQ8PZ/LkyYwbN06+s2+EP0wUkx6k\nNxWTW8ROa4+OR//BvIuZG+swXiu7/4UP4GZjI4435BmEs7N3wG1oBA4q1/tf/AdYjE0YKkuQLPKs\n77jZ0ICuTobTACQJ10GhqLxEYVKK6Jl0gv6nIorXT8IjJLrtSMgexNxYR3PNFZ54/3ucfeWZt9Dy\naw0XXhuJ+7DR2Ku6fqr6H2FtbUVfVkDk5jzch42+/w2C7EQxsZG1pZlzL/kT+u8Z+I596v43KKDy\nP1Np+P4ko5LkOTP52xf74j/rVQJf3CxLe3JrKP4a7YevMWbbRdl7ZML9icecTrBzVNlcSKqqqsjK\nyiI4OJji4mKgbfwgMDDwtnU6ISEhLFy40KbPUP/5r1w/+bFN93bEuV8AA6Yvtene7sjXO3IyFmMj\nWC023S90jigmCgkICCAyMpKrV6+2L2IzGAysXr1a4ci6Rm/LtzcSxUQheXl5HD9+nMWLF3Pp0iV8\nfX1pbm4mMTGR/Px8wsLCqK6uJiwsjMWLFzN8+MO9611vy7c3ElunK2TKlCm4uLhw7Ngxxo0bx5kz\nZ7Czs2PDhg2Eh4eTkJDApEmT2LRp0yPxi9Xb8u2NRM9EQZWVlVRUVGA0GvH09MRoNJKWlsb48eOB\nttm1hw8fRqVSdbgL2cOmt+Xb24ieiUJaW1tRq9WkpaWxceNG+vXrh729PW5ubsTHxzN8+PD2Yx0m\nT56sdLid1tvy7Y3Eq2EbWVuayV8ayL98VNMl7Vsslk5PDTfVV/P9xumM/Q+NLDFd/Fsk4euP4OIn\n//J1OfIFOLckgKi0yzi43PskAEF+4jGnm6xbtw5JknB3d2/fRBnatjLYtm1b+xYFmzZtAuCrr75i\n+vTpeHh4sGLFioduzcmtfGtqahg1atQjn68gikm3+PbbbyksLMTT0xOTyYS/vz+lpaWEhYWRm5vL\nzZs3KSwsZPTo0ahUbUcyODg4oFKpMJvNZGdn88MPP9zW5ty5c4mIiFAinfv6bb5ms5mysjKb8j1/\n/jzR0W17fvTkfIU2YsykG8TExODh4YGbmxszZsxg4sSJhIaG0tTURGlpKenp6WzduhVJkpg2bRon\nTpwgLCyMkJAQampqmDNnDmazmYSEBGJiYoiIiOjRv1i/zdfDw8PmfENDQx+KfIU2oph0k8GDBxMS\nEsITTzxBbm4ujY2NLFiwgNWrV+Pi4sILL7zA8uXLSUlJITIykiVLlrB9+3bGjBlDQUEB3t7e+Pn5\nMXPmTL7++mvKy8vv/6EKupUvYHO+JpPpoclXEMWk2zg7OxMcHMzHH3/MwoUL2bdvH2+//TYjR44E\nYPbs2cyYMYP9+/dz5coVLl68yLFjx7BarTg4OLBq1SoSEhKAts2XhwyRfxBUTrfy3bZtm835xsXF\nAQ9HvoJ4m2Mza0sz5/9tOOP2VCsdyl21/HqN4g3TGLv9f2Rp7+LfIonYcBRn3wBZ2usK+f8ayNjt\n3+Pg6qF0KL2O6JnYSJKs2Ktcab5RoXQod6W/UkjLLz/L1p6jhy/68u9ka09uzTcq2xb62YmvtRJE\nz6STdJpTNJUVAJ3fz8RsNnd4cp0t7Bwc8AqfiHuQvEc8GK+V8WvhV0gWsyzttbS0tL/R6QwJCfeg\nx/GO+HPngxJsIopJD7Jt2zaCg4OZPXu20qF0i9zcXDQaDWvWrFE6FEEGoj/YQxiNRoqKijh+/LjS\noXSb48ePo9FoMBgMSociyED0THqIq1ev4ubmhqurK0ajEbVarXRIXerGjRu4uLhgMBgwmUwPdLaQ\n0LOJYtKDiN3phYeZeMwRBEEWopgIgiALUUwEQZCFKCaCIMhCFBNBEGQhiokgCLIQxaQHcXfvfVsN\nenrKc7ayoDwxz0QQBFmInokgCLIQxUQQBFmIYqKAiooKZs+ezalTp7hw4QJTp05lxowZ7Qd679+/\nn5UrVyocpbzee+89oO184ZdeeomsrCwA9Ho9AFu2bMFisWA2mzGZTIrFKdhOFBMF6PV6TCYTubm5\nxMfHM2zYMNRqNfHx8QDEx8ej0Wi4ePGiwpHKY9euXRw4cICkpCSmTZvGrFmzyMjIYOvWrbzyyisA\nFBYWYrVa+eabb/joo48UjliwhTjqQgFVVVX4+voyc+ZM9Ho9arWap556irNnzwJQU1OD1WolOTmZ\nzz//XOFoO8/Hx4eRI0cyYMAApk+fTklJCbNmzaK0tJSmpia0Wm37v1VVVUqHK9hIFBMFODg4oNPp\nOHr0KKWlpVRUVNDS0tLevd+xYweJiYkUFhaSlZXF/PnzFY64c/Ly8tBqtQwfPpzKykqcnJxwc3Oj\nT58+xMXFkZOTQ3V1NYcPH6axsZEJEyYoHbJgA/GYowBfX1/8/PzQ6/X8/s38oUOHKC4uZuLEiSxf\nvpydO3ei1WoVilQeKSkphIeHExQUBIAkSeh0OgDc3Nx44403GDFiBCtXrmTdunWcPHlSyXAFG4li\nogC1Ws2YMWNYunQp0dHRBAcHM2/ePBoaGkhLSyMzMxMAJycndu/eTXx8PD/++KPCUdsuOzub/Px8\niouLMZvNSJLUfuTFrQHYW6xWKw0NDUqEKXSSmLSmkOLiYvz8/MjJySEsLAyz2YxKpSIqKgpXV9fb\nrr1+/TpqtRo7u85vWq2UtWvXMmfOHPbu3UtQUBAWiwVoO3vY2dkZo9GIi4sLra2t+Pn5kZqaqnDE\nwh8mCYKMCgoKJKPRKKWnp0uSJEmZmZlSbW1t+8+rqqqk1NRUqa6uTtq6davU3Nx82/3r16+Xmpqa\nJEmSpOrqaunQoUPSrl27pGvXrklz586VTp8+LZ0+fVoqLS3tvqSEByIGYAXZ7Nu3D4PBwNixYzl/\n/jxDhw5l+/bt5OXlMXjwYJKSkkhOTub5559HpVJRUlLCu+++y9SpU9mzZw86nY7CwkJqa2sBeO65\n58jJyWHSpEm8+uqrDB06lKKiInbt2kVGRobC2Qq/J4qJIJvMzExycnLQaDRMmjSJuro6pkyZQkJC\nAv379+fq1avs3r2b8ePHk5iYyMmTJ3FwcGDQoEHs2LGDp59+mmPHjjFs2DAAvvzyS0pLSzEYDCxZ\nsoT09HRKSkr45JNPGDFiBEaj8Y5HQkE5opgIsjCbzTg7O2NnZ4ePjw/Z2dk888wzZGVlodPpcHR0\nxMnJidjYWKKionj22WcpLy9n586dAGzatAmNRnPbWMkHH3zAqlWrCAoKYt++fXzxxRckJiai0Wjo\n168flZWVjB49WqmUhd8RxUSQnb+/Pz4+PixatIgDBw5QW1vL+++/j4+PD2+++Sa+vr64u7ujVqux\nWCycPXuWgoIC4uLi+PDDD9vbOXjwII6Ojri7u+Pq6sqyZcuIjo7mrbfeIiYm5o7X6oKyRDERZOHk\n5ITJZEKSJMxmM+Xl5WRlZXHw4EG0Wi3r1q0jPT39tnsWL15Meno63t7eZGZmMm/ePPLz89t/rlKp\nWLJkCVFRUVy6dAmtVktqaiopKSkMHDgQLy+v7k5TuAdRTATZzJ8/n7179+Lo6Mjrr79OUVERK1as\noLW1FScnpzvmysTGxrJ582Y+++wzPD09aW5u5vLly+0/X7RoEWlpaZjNZs6cOcOQIUPYs2cP/v7+\nlJSUtM9VEXoGMc9EEARZiBmwgiDIQhQTQRBk4fjTTz8pHYMgCA+5Rp2O/wUsqgYgG2sJfwAAAABJ\nRU5ErkJggg==\n"
    }
   },
   "cell_type": "markdown",
   "id": "00edde14-0c43-47ac-9d9a-514e15813fc3",
   "metadata": {},
   "source": [
    "# transformer\n",
    "\n",
    "![transformer结构图.png](attachment:ad44b9b3-fffd-465e-813e-1fa785bafb31.png)\n",
    "\n",
    "基于Transformer 的编码器和解码器结构如图2.1 所示\n",
    "左侧和右侧分别对应着编码器（Encoder）和解码器（Decoder）结构，它们均由若干个基本的Transformer 块（Block）组成（对应图中的灰色框）\n",
    "每个Transformer 块都接收一个向量序列作为输入$\\{x_i\\}^t_{i=1}$ ，并输出一个等长的向量序列作为输出Y\n",
    "$\\{y_i\\}^t_{i=1}$ 是当前Transformer 块对输入 Xi 进一步整合其上下文语义后对应的输出。\n",
    "\n",
    "## 几个transformer中的机制概念\n",
    "\n",
    "注意力层：使用多头注意力（Multi-Head Attention）机制整合上下文语义，它使得序列中任意两个单词之间的依赖关系可以直接被建模而不基于传统的循环结构，从而更好地解决文本的长程依赖问题。\n",
    "\n",
    "位置感知前馈层（Position-wise FFN）：通过全连接层对输入文本序列中的每个单词表示进行更复杂的变换。\n",
    "\n",
    "残差连接：对应图中的Add 部分。它是一条分别作用在上述两个子层中的直连通路，被用于连接两个子层的输入与输出，使信息流动更高效，有利于模型的优化。\n",
    "\n",
    "层归一化：对应图中的Norm 部分。它作用于上述两个子层的输出表示序列，对表示序列进行层归一化操作，同样起到稳定优化的作用。\n",
    "\n",
    "## 嵌入表示层\n",
    "\n",
    "对于输入文本序列，先通过输入嵌入层（Input Embedding）将每个单词转换为其相对应的向量表示。通常，直接对每个单词创建一个向量表示。\n",
    "由于Transformer结构不再使用基于循环的方式建模文本输入，序列中不再有任何信息能够提示模型单词之间的相对位置关系。在送入编码器端建模其上下文语义之前，一个非常重要的操作是在词嵌入中加入位置编码（Positional Encoding）\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3fa6bac9-591a-431d-9eff-ec4a4e0c2fd5",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'nn' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[4], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28;01mclass\u001b[39;00m \u001b[38;5;21;01mPositionEncoder\u001b[39;00m(\u001b[43mnn\u001b[49m\u001b[38;5;241m.\u001b[39mModule):\n\u001b[0;32m      2\u001b[0m     \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m,d_model,max_seq_len \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m80\u001b[39m):\n\u001b[0;32m      3\u001b[0m         \u001b[38;5;28msuper\u001b[39m()\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__init__\u001b[39m()\n",
      "\u001b[1;31mNameError\u001b[0m: name 'nn' is not defined"
     ]
    }
   ],
   "source": [
    "class PositionEncoder(nn.Module):\n",
    "    def __init__(self,d_model,max_seq_len = 80):\n",
    "        super().__init__()\n",
    "        self.model = d_model\n",
    "\n",
    "        # 根据pos和i创建一个常量PE矩阵\n",
    "        pe = torch.zeros(max_seq_len, d_model)\n",
    "        for pos in range(max_seq_len):\n",
    "            for i in range(0,d_model,2):\n",
    "                pe[pos,i] = math.sin(pos / (10000 ** (i/d_model)))\n",
    "\n",
    "        pe = pe.unsqueeze(0)\n",
    "        self.register_buffer('pe',pe)\n",
    "\n",
    "    def forward(self,x):\n",
    "        # 使得单词嵌入表示相对大一些\n",
    "        x = x*math.sqrt(self.d_model)\n",
    "        # 增加位置常量到单词嵌入表示中\n",
    "        seg_len=x.size(1)\n",
    "        x = x + Variable(self.pe[:,:seq_len], requires_grad=False)\n",
    "        return x"
   ]
  },
  {
   "attachments": {
    "caff02a6-bcee-4ccd-9247-abbb111f0ca0.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAARMAAAD3CAYAAAA+C7CYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAOxAAADsQBlSsOGwAAIABJREFUeJzs3Xd4VMXewPHv9t30HhISIJCQ0Htv0gVEqnSlWFERvPjq\nFcXeALsIoqKCgIB4EREbIFVCkxJaQhJSgPTetu++f6wEIiGNTTbAfJ6Hh7Nn58yZs8n+MmdmzozE\narVaEQRBuElSRxdAEITbgwgmgiDYhQgmgiDYhdzRBRBuHzExMUydOpXBgwdTUFBAeno6qampbNmy\nheeee46PPvqII0eOcPbsWZ544gkmTpyIVCqlWbNmBAUFERMTQ1BQEOnp6cyZM4f169ezYcMGunXr\nRqNGjbh06RIPPfQQ7dq1c/SlCuUQNRPBbsLCwggJCSEiIqL09ciRI9m1axcAcXFxXLx4kYyMDGJj\nYzEYDGRnZxMUFMSePXvIyMggJyeHgIAAgoKC0Gg06PV6srOzadu2LQUFBYSFhTnyEoUKiJqJYDdR\nUVGkp6fzxx9/YDabSU9Pp1u3brRo0QI3Nzd27NhBfHw8aWlpbNmyhYiICFatWoWTkxMymQy5XM6J\nEydQKBT069ePU6dO4efnh4+PD3v37sXLywuDwYCTk5OjL1Uoh+yVV155xdGFEG4PXl5ehIeHk5OT\nQ3Z2NhaLhQYNGhAeHs7q1at56623cHZ2xtXVlXnz5rF582bCwsKYPn06Go2G7t27k5WVxaBBg+jQ\noQMNGjRAJpORm5tLREQEvXv3Jjc3l+DgYEdfqlAOUTMR7CYmJobPP/8crVaL0WhEIpGg0WiwWq3k\n5uZelz4nJ4euXbuSkZHBt99+i7OzM926dSM/P5+lS5fi7OxMVlYWycnJNG7cmAMHDtCnTx8HXJlQ\nFRIxaE2wp7i4OPbs2cPx48fR6/XodDqWLVvG4sWLycjIwGQyYbFYUCqVHDt2jGnTpjFo0CAMBgNf\nffUV77//PrNnz2bmzJn06tWLs2fP8uuvvzJ//nxHX5pQCRFMBLuyWq1YrVYkEgkSicRuedorL6H2\niGAiCIJdiK5hwWEsFgs5OTmOLoZgJyKYCA6zYcMG3n//fUcXQ7ATEUwEh7BYLJw8eZKCggJRO7lN\niDYTwSGys7NxdXUlPz8fk8lEQECAo4sk3CQRTASHyszMxNfX19HFEOxA3OYIgmAXYgSsUC3FuiJM\nZqPdxn0UafNRlSjskheAm5OH3fISqkcEE6HKYi+fYcFXM3F1csdisTi6ONcp1hXy5L0v06/dCEcX\n5Y4kgolQZQu+msnMEfNo06yzo4tSrrzCbF77ei6dw/virHZ1dHHuOKLNRKgybzc/Gvo2rlLaQ38d\n4dXn37puf15ePo8+MIdLyZfJy8u3a/k8XL1RKlRIpTK75itUjQgmgt2ZzWaWvr+C+QueAiAjPZOz\np6IB2LRuM02aNiY2Jo7XX3iHi0mXOH70JAaDgZPHT2E0GtHp9Bz66wipl9PIy80jKzOb7CwxFqW+\nE7c5gt3JZDJCmjVmyRsfMnbivaz9ZiOhzZvy195ItCVaJEgoLi5Br9Mz95H/Y+Ddd7H49Q8YOPQu\n9u06QJduHbkQl0BgUADvvPoeBr2BV9550dGXJVRC1EyEWvHa4oWMGH03kfsOM3XGBB6cPZ2o46dp\nHhFGu45taN22Jc3CQujaozOTH7iP0OZNmTZzEkkJycTFXiAxIZngxkFoS3QEBgXi5i7aQOo7MWhN\nqLJHPxzBk+NfxMPVu9K0+XkFWK0WPDw9MBqNGA1GnJydMBqNACgUCvLzClCplKjUKnRaHRonDdoS\nLWqNmqLCYhQKOQqlArPZDIBSqaz0vP/5eBqrnv0TJ5XLzV2sUG3iNkeoskCfJiz731uMvWs6FmvV\nuoZTrm1jzaydcl0RnRQFgFKurt0TCeUSNROhygwmPfOW3YeXmz8qRf37wsqkMh4fuRAPl8prToL9\niWAiOExqaioJCQn07NnT0UUR7EA0wAoO88EHH7BkyRJHF0OwE1EzERwiOzubo0ePAtC4cePShbuE\nW5cIJoJDWCwWpFIpmZmZeHt7I5WKSvKtTvwEBYe4NniIQHJ7ED9FQRDsQowzEaol/lIOBUV6u81n\nkpeXh0eO2S55yWVSWof62SUvofpEMBGq7Oe9Mby4dBctm/lhsdirqc0K2CcwRSdk8sDIdvzn/h52\nyU+oHtEAK1RZ+wmf8c7Tw2gWXD8HhekNJqb9dz17v56Jm7PK0cW544g2E6HK/L1dcK3ml3Trj9+X\nbq/49D17F6kMlVKOWilHLhO/1o4gPnWhVu3a8QtXKr8Dh4zAoNdz6WISp6OOYzAYuHwpmeSkBADS\nUlM4dvQgFouFSxeTyM3NdmTRhWoSbSZCnVn+yRL+8+zLzJ/zIKPHT+aXrT9gNpvRlpQw4+EnOBS5\njzNRJ0iIjyUp8QJTHnjY0UUWqkHUTIRa98Hi11jzzeelr7v16MPkaQ+Snp5KZmY6Xt4+gITft/2I\nm7s7F5MTAWgQEOiYAgs1IhpghSob8ti3vPrEEHw8nat8TElJMUaDAZlMhkwuR63WoNfpUGs0lJQU\nI5PKMFvMODk5U1xchEqlxmwyYbVaUWs01S7jff9Zw/5vZuHiVPncJ4J9idscoco6tghg9uubmTm6\nM5Z6+Dfo4MkkAJw19luHR6g6UTMRqsxisfLOV/sxW6yoFPVvBnipVMITE7ugUYtg4ggimAgOc/78\neWJiYhg5cqSjiyLYgWiAFRxm6dKlrFixwtHFEOxE1EwEh8jIyODy5cu2hla1mpYtWzq6SMJNEsFE\ncKjMzEx8fX0dXQzBDkRvjlCpgoICTp06RWpqKlqt1u55u7m52TXPqggODqZ169b4+PjU+blvV6Jm\nIlTqzTffJDw8nKCgINRqNfb8lcnNzcXT09Nu+VVVcnIyUVFRdO3alaFDh9b5+W9HIpgIFYqMjKRR\no0a1VnvIyspyWO1Ar9ezatUqQkNDGTVqlEPKcDsRvTnCDaWmprJjxw6H3IbUBZVKxaxZs7hw4QLn\nzp1zdHFueaLNRLihI0eO0L1799LXJl02cdvHYdLZb2k+q9VKmj1mbbNa0BddpuWYw2g8qz7TvVwu\np1evXhw+fJgWLVrcfDnuYCKYCDeUnJzM3XffXfr65LoQGoT0pHGXR7FWcXnQupSfGcfZzV1pf38q\nMkXVnx8KCQlh+/btpTPmCzUjPjnhhrRaLZprHrZTOgfQuOUINC6+OLn6l/57YuGvpdtxF438uvcy\n//sjASdX/zLbTq7+fPzNcXYfzuDBZ3/ifJKhdP/Q+1fzn9e34+Tqz0PPbcXJ1Z+TMUWonX2RKb3I\nyJNxMd3KxXQr8ZdMGCyumCXuZcoR0LQXUpkasLJx40ZycnKqdJ1qtRq9Xm/XhuU7kaiZCDfl0LEE\nTCYLefklrN9yhLx8LadjLnN3/1Z8tnoP+w/HcXf/VqXpAxu489hza1n84jg2/3qc9q2DAQgO9CSw\ngQc799naLgqLdMQnZbJqYyTPPzWMybO/5O0FY5j30kaemT2YiY9+Tse2jViz9MHryrRixQpOnjpP\nSkrKDcs9b948O38SgqiZCDdl1cZIMnMKWbf5MHsiz/PM7MGMGdYBgD2R50u3AQwGE1t+O0nSkbc5\nceYix04ll8nrv0/ezSdf7UKrM3LkRCLTxnUnN78EgN5dQ7mrZzgtmwcwbVx32rUKusGk1lby8vLQ\narVkZ2eX++/IkSO19nncyUTNRKgxrdZASnoeW1c9yb3TP2XiqM489eIGCot0DO3fkqF3teLHX08w\ntL9tqLxCIUOhkLFw0RZy8oqR/WuuVqVSzvNzhjHygaV8t+whFi7aQpf2TUrfA9uTwcB1x14l4bnn\nnmPNuh8YOXIkXl5e16XYtm2bfT4AoQwxzkS4oSVLljBlypTSruFTGyLoeNcc1M62L6jFYkGnM+Hk\npKSoWIeLs5rCIh0atQKLxYpSKcdkMpduX5GbV4ynhzNGoxnFP1MZFJfocXayTVZdUKjFzVVDfoEW\ndzcNVqsVnc6IRqMsTVdSYsCKtfSYK/b+8BRtpyQgU7jc8Lq2bdvGiBEjrrvWl19+GZms/k2tcKsQ\nNROhyuQqD1IT9hPcfDBWbL05CjkYDUZUCjAailErwWoxIwGMBj1AmW0AFydbWgCjwbZPKQejwQSA\nRmV730l9NZ1cZjvPlXSKf35zrxwDUJCdgMV89TxC3RLBRKiyZoN/4NSGFiRH/4FUVv/WpZHKNUTc\n82eFtRKh9ohgIlSZ0rkhnWYV2DVP8dTw7UMEE6HKDp9OZf4Hf9KhXYi9VvT8Z3XQ2JvORiqR8Pfx\nCzw5qRMTB1d9BKxgPyKYCFU2/4M/uffe7jRp4u/oopSrQ6fmfLV2J/f2DUWjEr/adU2MMxGqzNvD\nCS8vV0cX44acnFSYTBYxktVBRPgWasWWr96l590TOLj9f4ycPg+T0cCPK5dgtVjwbhBEQW4WcoWS\n5u26YTIaAdi37TsUShWjZs5n66oPuWv0A6QnxwPQse8wFCq1Iy9JqIQIJoLdGfRazp88hJOLG7qS\nIr77+GV6DBlLZkoyTi5uXI7cibd/QywWC7+fO4Gbpw8KlRqNkwv+wU05eWA7RoOOQ9v/R9dBYzi2\n5xciOvbCXQSTek3c5gh2F/n7D8gVClw8vLlv9osU5Wfj4dMA38BGmIwGigpyMeh1+AQE06xVR5zd\nPPH0DUChVGPU63Hx8MbVwxtdSTH+QSG4eXo7+pKEKhDBRLC7iI69aBLRnrY9BpKScJ6QiPZ4+Phj\n0GkJDAmnU78RBIaEYzToadN9IEa9jrSkOPJzMog+cQDfwEZYreDmKeZnvZWI2xzB7vyDQnBx82DX\n/1ahLSlEqVIT+fsmFCo1aqerA8qK8rLx9G2Af3BTigvz8PZvSEjLDpw9so/hU5/g1KFd5OdkoHFx\nR64UawfXdyKYCLVi8IRHqpx2wNgZZV4HNAoFoNewCUgkktLXQv0mgolgd7qSYkoK89j85WJMRgMT\nn3wFpVrDni3f0r7PUM4c3kP8mb9p3q47GZcScHK5OsesXqdl/GML+O7jl8m4dIEuA+5FqdLQuf89\nDrwioSpEMBHsLu1iPFEHduAb2BgPH3+2f/8Ffg2bcDkhhhN//UGbbgMw6LScPxGJVCqj17AJZKYk\n4eXfkD82rODs0b04ubiSn5OJxWIhNuqwCCa3ANEAK9idu5cvly6cQ+3kQnbaJaRSGT4BwTi7etBt\n0BgUKjWevgE0Dm+Lp18gX7/9NEUFufz09ft4+TXE0zeAC2eP0fmuEfy69lMCQ5pj0Nt38S/B/kQw\nEepEqy79cPfxw9u/IbqSQkxGAxdjzyCVSbFYLLi4e1FSlI9EKiEvOwOrFeLPHKP7kLFEHdhBfrb9\nZsQXaocIJkKtUChUqJ2cCWgShsViBiApJorzJw8SGBKOtriApq06cuHMMZ5avJroYwcY+/BzpCXH\nE9AolJ53jye8fQ/ueWAugSHh+AY2cvAVCZURbSZCtUiqsMaNp28AD7+0tPR198FjAZj92uel+7r0\nHwnAwHGzAJj81KsAhLXtVuYYgHGPPl/l8kmlEsSjOY4haiZClXm6qkhKynB0MW4oMzMfnd5UOk+s\nULdEzUSospce6cUDC38mNiYZpaJ+/epIpRJiL6Tz1pN9xfQDDiI+daHKQgLd2fbReKITc7BSv+4l\nrFYrQX7dCPKvv1Mk3O5EMBGqxc1FRdfWAXbJKzExkfj4eAYOHGiX/ATHEsFEcJjly5eTkJAggslt\nQgQTwSGysrIYO9bWYxMdHU1EhJi39VYngongED4+Pvj4+JCVlYWPj5hq4HYggkk9kpOTwy+//OKQ\nc0+dOrVKY0jsbe7cuahUdbsGT0pKynUr+gk3TwSTembatGkOOa+jJmEeOHAgrq6iB+Z2IIKJUC2G\n4hQsxiK7rZszbEBrPD097JKXRCJH5dbULnkJ1SeCiVBl+Rf/IG77eDRuQfaryVit5Njp9kpXeJmg\nLq/j32auXfITqkcEE6HK4raPp33/+Xj4Nnd0UcplMmrZv3ke3s0fQK7ydHRx7jgimFwjJyeHqKgo\nLl++jFZb9/NnFBcX4+zsbNc8VSoVfn5+tGnThsDAwJvKS+kcgMrJ67r9x08l06HN1ad6I4/Gk1eg\nZdiA1mW2AVLS8vhs9V76dAtlcL+Wpce8u/wPHpzcC08PZ77bfJjJY7pWu3xyhQapTI1EqgBAp9Oh\nVovlMeqKCCb/2L17NwcOHKBVq1ZERESg0WjqvFEyPz8fd3d3u+ap1+vJyMjg66+/5oUXXrBr3ld8\nvPJPvv5wBgBRZy+Rkp6PVmcAKLMNsPr7g3TvFMJX6//C19uV9q2DAdjy+0niEjL4bPE0tm6PYvKY\nrkQejadbxxCMRjOpGfno9SYADEYzwYGeyGRSXF3KDxZLlizhiSeeqJXrFcp3xweTixcv8s033zB7\n9mw6derk0LLk5OTg5XX9X/6bFRYWRq9evVi+fDndunWjY8eOdsv78f+uIzUjn2Xf7GbfoVi+/WQW\nsQm2J4snz/6C8fdc/UwNBhNHTiTy3JND2XswloWLt7B19ZMABAd68vEbkxg9cxkSiYQ/90czoHcE\nEx/9nMULx/Hp17tZ8tJ4pjz+JeuWPcTk2V8gkUhYt+yh68p07O9jZGdnc+LEiXLLfPDgQf7zn//Y\n7TMQbO74YHL48GH69OlT52MdHKF3797s37+fDh062GVMiVZrICU9j62rnuTe6Z8ycVRnnnpxA4VF\nOob2b8nQu1rx468nGNrfdjujUMhQKGQsXLSFnLxiZLKyM2AolXKenzOMkQ8s5btlD7Fw0Ra6tG9S\n+h5QOr3Av4+9VsdOHVGo3WjUqFG5wTk3N/emr1243h0fTJKSkujcuXONjtXlx3Hhz6lYzTq7lMVq\ntZJqp54Ni1mHoTiVdlMSkKttK+I1btyYH374AbPZjFxegx+91YJEcvVLrFLJWffpQygUMr7/4hFc\nnNWMG9ERjVqBxWJFqZQzbVw3LBbb7aJEImH9Zw+Tm1eMp4czRqO5NK8v3r0fgG4dQ4iLfAO1WsEz\ns4fg7ma73Xxx7nAAViy2jcP5fPH95T65bDHrwGqhffv21b8+4abc8cGkpKSkRo2eFpOWMz90JKzD\nJDz8wmuhZDdHIpGSdO5X4n4fRcSo/f/sk6DRaCgpKcHNza2SHK6n8W7HwZ8X0KLbTKxWS+n+gn/+\nL/pnmtbiKuSVlnf9vsJrtkuybf9rc67uyy8n3bXbWZdPAiBVuCDUvTs+mMDNjP6U0DCs/3V7M7IK\n8PO5+mX96ru/GNA7nCbBtmdQEi9mlW6fOH2R73/+m8mju9A6oiFga1tY9OnvvDhvOBKJpMa9G03b\njuXojiU1ubByNRu4ntg/xpKaloZUrrFLnka9AYXKPqv1STQRtJu6pUztSag7IphUU05ODrt27WLM\nqBFIpNd/fLl5xaz+/iDPzB6C1Wrl6MkkLqXmotOZSl+7Ol/tgfj0m93MmNCD1d8fpG/3MO4Z3BaD\n0cTna/bi5eHEEzP7l/ZupGXk08DPneycIgxGM/mFWtQqOemZhXRoHYxEIkGhkJXmfW3twR4kUjnN\n7/7JrnlmZmbi6+tr1zwFxxDB5F/MZjPLly+vMM25c+fIykyhSzm9kt/9eIQ9ked5ZvYQ5i7cQKe2\njflt1xnuG9mp9HXXDk1K0/t4uTDliZX8vm4uCxdv4Z7BbQEYPrANB45eYNRQ273/kROJfL5mHy2b\nB9A4yIuNP/3Ng5N78er7P3PPoLa88M6PDOwdwfNPDbPbZ/FvFqOOpGUPUhL/N1aLufIDqijdDnlY\nTbbu5xbvHkfuYv8eMaFyIpj8i9Fo5NVXXyUzs/x1WnJzc9m8eTMzHpjCiTWvlHnv1LnLXEjK4v1X\n7mP2c2txc1UTl5CBSilHggSNWkFcQgbdOoQAkJNbzKlzl5kypisfr/wTk7lsTWLtpw+y4tu9ZOUU\n4eKsorhET9NGPpgtFibc24nmzfzp16M5D0/tTWpGPvFJtbu2TPSzXWj99NeoPBvU6nluxtEXehGx\n6AgytWg3qWsimFSTp6cns2bNwmLWX9e9Gt7Mn7eeH41SKWfJS+NwcVZTWKTjhbnDUSplLHpxHIVF\nOpydbG0EXp7ObF39BAWFOtzdNBSX6AFwcVbzwSsTAHhkWh+mju2Ki7OaVR/NRKGQlfaCyOVSXpw7\nHLVawdvPj7murBIAq/1qEFK1Cze7jsSpc+cp0Wrp1rEdAOmZWfh6e3HkxCnatghHo7nJEatWK3Z7\nClGoFhFMashqMWIxG8hJP4ebl62mcWWFBZPRhFoJJqMOjQrAjMVsxmIGjQosZgPX3iU4aySYjDpU\nCtsxAMprtq/kJQFMRmPpV8VsAoUczCY9SsWVc9uOkUgkpCcdwqi7pjvEwZZ/8x27/jpI64gwPljx\nDcGBDZg05h6ef/M9XJ2daf5/TW4+mAgOI4JJDckULoQP/52oX4YikamQ1Le/hhIpcpUnrSecdnRJ\nAEjLyCTA35eQRkFkZufQtUNbnn50Bg/Pf5HHZ05l09bfUNmpV0dwDBFMboJLgx50mlVQecIqqq3h\n9PVBanomeyOPEJuQRNLFyyjkCmY/+zIjBt3Frzv3cj4+kd1/HUYmk9KpXWt8vMRTv7caEUxuglZv\n4te/LmAwmuySn8FgQKlMs0teaqWce/qGIq9g2HldatOiOYXFxfTv1Y2QRkGcOH2OAb274+/ng8Vi\nYdnXa5k6biSNggJR1GR0ruBw4qdWQyaThSGPbyCksS+e3tUfTXpj9hmafz7mMu+tOcKO5RNRKR3/\nYzYYjQzq05NT52KIibvAgnmP0TioIc+/8R5FJcXs3bKWles2ceTEKV6e/6SjiyvUgON/y25R6347\nS+fOYfTs2bLyxA7Qu3dr9u87zeZdsUwa2sJh5bicmk7DAH9OR8cyqG9PJo4eXvqe1Wrl7Rfnl75+\n58VnOHs+jqycXIIb2mehL6HuiGBSB7JSL7L/l/WMfvD/WLX4/5j+7BJ2b/kWo16LTluCrqQIZzcP\n2vccQl62bQjXoR0/YjYZGDzhEfb+tIYB42ah0jgTf+oIDZu1IKjprbHOzEdfrKJZk0acjj5P7IUk\n+nTvzP3j7+XLtZs4ExNHanoG0yeOYdr4e0m8eJlfdu6hYQN/2rWKoFV4mKOLL1SDCCZ1YMemL/H2\nb8imz97CydWdwzu3kJV6EaNex+XEGHwaBFNckEdhfg6nDu5EV1JMcUEebbr3R1dcSGF+Dps+e5MB\nY2YQGNKcY3t+uWWCSce2rRg3YgixCUns2HuAR++fRNTZaC4kXSS/sBAnJw1nY+I4ez6OM9FxHD1x\nGtqDwWAUweQWUz9a525jupJiDm3fTIvOfTHotdw3+0WCQ1sS2KQ5Rfk56EuKMRkNmIwGPLz98PYP\nwt3bj4597+bY3l+RK9WEtu6CxWympDAf/+Bba/b1Rg0D2fzLdpZ9vY7f/tzHu8tX0qVDW1QqJX27\nd6Fv9y54ebrTpX1bZkwai5eHO/Mfm8WMSWMdXXShmkQwqWVypZKB4x9EJpdjMZswGQ0ENA7DzcsH\nZzcPOvYbTsOmEWiLC/HyC8Rg0KEtKuTs0b04ubpz9vAewtp2pSA3i6DQ+tk+U5Fvv/+RCaOGM++R\n6dwzuD8vzJuN2WzG2UmDwWDA28ujdJ7WzOwcdHo9Umn9/bXctWtX6fb58+fJyak/gwIdTdzm1DK5\nXIF3gyAAOvcfSfL50zRtZZs2MSUplvwc2/M0hblZKFRqTAYDIS3bo1SpSb+YQGibzpw/eZChkx5l\n94+rCWrmuMbUmmjTIpyXF39MTl4e8YkXSc/MwtPDjWED+rLrr0OcPB1Ny/BQfvr9T1au+56P3niR\n33ftY9/Bo7zx/NOOLv51Bg0axNatWxk+fDhr1qxh4MCB9OvXz9HFqhdEMKkDao0z0X//RUFuJmaz\nicTok5QUFdCycx80zrZu5ayUJI7v+512vQZzfN9vAAye8DBxUUdxcnWn++CxhLXtisloJOHccUde\nTrXcf98oXF3KTj5lMpmQy+W0aRGOk0aNl6cH6ZlZDOjdDRdnZ1ydnQkNaeygEl9lsVhYsGABr7/+\nOkuWLOGhhx5i/PjxvP766/Tt29fRxat3RDCpAx363F2t9I2btyndbtdrUOm2t7+thuMfFGKfglWT\nRK7AYjJUnvAa/w4kQOmUkUGBV58+9ve9uni5t5cH3l7VX+XPajZhMWiRyGSVJ64CqVTKiBEj6NSp\nE4899hh+fn54eHgwefJkFixYgIeHfVYivF2IYFLLDHotOzd9hdGgRyKV4uHth1Su4NjubUhlMuQK\nJQa9jiYR7WgQ3BRv/yAuXThHx77D2LFpJYEhERzf9ytunr5cij+Li7snU+a+gbu3X51fS/PX9pCy\n8RVK4o8iVdl3fZ+bZ0UVEEarpbFI5fZ5xsdisaDT6YiKiuLQoUPk5dnmmhw9ejQ9evQgIiKCgQMH\n2uVctwMRTGqZUqUh43IiAY3DOHdsPy069sbTLxCNiysRHXtTlJ9DVupFlCoNSCTs+nEVTVt14ovX\n59C8XTckgJOLGyEt2hHWpgt/7/0Fva7EIdcikUgInPhqmX16vZ5t27Yxdmz1e18yMzNrPKG3yWQi\nMTGR0NDQah9bVVKplMGDBwPQrVs3AN577z0A/P39SUpKQqOxz/SVt4P622x+G1OpNTQJb0fXgfdy\n+UI0RoMeo942jF4qlWEy6DAZ9Di5eiCRSrlw5hi5mWns3rIaZ1d3Th7Y7uAruGrVqlV89NFHNZpH\n95NPPuGDDz6o0XnXrVtX+sWuSy4uVyddcnNzQ6FQ1HkZ6isRTOqAVCpF7eSMXK7AL6gJAGaziV/X\nLiO0TRfyslJp0qIdao0zXQeNwmqFcY+9QFpyPBaLhXGPLcBqMRPSogNdB46iQXDt/TWuDr1ej7e3\nN08++SSHDh2q1rE5OTn06NGDadOmcf78+WodazQayc7OplGjRiQnJ1frWKH2iGBSBxo0akZgSDiN\nw9vSILhvyFtnAAAgAElEQVQZYW274hcUQvP23SnIzeKhFz4hMyUZi8VCqy79GDLxERRKJVarheZt\nu2Ixm7lr1ANYrVYO7/wJpbp+VK3lcjnjxo2jadOmdOnSpVrHuru7M2zYMLp06UKzZs2qdaxOp+Pp\np59m6tSpNVqyQ6gdos2khqyAtIoLZg2e8AgAoa2vtg106D0UgBYdewHQf/QDZY4JDm3FlLmvA+Dh\n4w/A+McWVKuMUqmkVtdLll3TayKrZg/KzRzr6upaui16VOoPUTOpIU9XFTExFx1djAod/TsOD1cx\nDaJQN0TNpIbu7RfGvf3COB2Xic5gn0mbL126hIeHR5lGvprSqOXsWzm1yum/+uqr6/YdPny4dBXA\nypw5c6Za5btiwoQJNTpOqH9EMLlJrUPtt4DU5x++RJMmTViwoHq3M/bw4IMPXrcvJyeHyZMni3YJ\noUpEMKknEhISGDBgAAD5+fm4u7s7uESCUD0imNQTgYGBDBkyBC8vL/R6vaOLIwjVJoJJPaFSqSgu\nLi7dvlN06tSp2r051x57bc9OVSUmJjJixIganVO4MRFMBIdq1qwZzs41e84nPDy8RoE3JMQxD0re\n7kQwERzq008/xdfXfo3YguOIYPIvUqmU9u3blz7gVZdMJlOZmbzqG6M2g7jf78Wote8C6al2ycWK\nUZtFi1H7cPJuZ5ccheoRweRflEol27c75kG6+j4FYNR3oTQMG0CjiBlYrRZHF+c6BdkXOLulD+3v\nT0GmuPmxOnZhMWE16rCa9Pyy+QcupqTgUUlXe3pmFkrvIHx8fCpMl5mZidVqxc+v4ukoUlNTCQio\neOmQ/Px8CgsLCQoKqjBdUVERbm5ujB8//rr3RDARqkyu9iQ4fDAqTdlu6+OnkunQplHp68ij8eQV\naBk2oHWZbYCUtDw+W72XPt1CGdzv6py27y7/gwcn98LTw5nvNh9m8piu1S6f2qkTMUecsD3s4CBW\nC1aTHqtRDyY91mtWqD98/CQNfH1o3aJ5hVmciYnFy9mHNm3aVJjup59+wmAwVDqnyvHjxxkyZEiF\naSIjI0lMTGTYsGEVpouKimL37t0imAg3RypTl1sj+Xjln3z94QwAos5eIiU9H63ONiPbtdsAq78/\nSPdOIXy1/i98vV1p3zoYgC2/nyQuIYPPFk9j6/YoJo/pSuTReLp1DMFoNJOakY9eb1uG1WA0Exzo\niUwmxdWl7OMCtvJJWLt2LcOGDauztZutZiNWox6rvhBu8DyUk0ZDcMMAWoRV/GBjAz9f/IKCaNGi\n4vl+//77b/R6faXp/P39K02TkpJCSkpKpekKCws5duxYue+JYCLclEPHEjCZLOTll7B+yxHy8rWc\njrnM3f1b8dnqPew/HMfd/VuVpg9s4M5jz61l8Yvj2Pzr8dJgEhzoSWADD3buOwdAYZGO+KRMVm2M\n5PmnhjF59pe8vWAM817ayDOzBzPx0c/p2LYRa5aWHblborPy/gcfEx0TX+H0BM8///zNXbjVitWk\nK7cGcqcSD/oJN2XVxkgycwpZt/kweyLP88zswYwZ1gGAPZHnS7cBDAYTW347SdKRtzlx5iLHTpX9\nsv/3ybv55KtdaHVGjpxIZNq47uTm22aV6901lLt6htOyeQDTxnWnXasgLJbyawBXnpS2Wq3l/qvu\n/CnXsuiKsBRlY85PxVKci9VQIgLJP26bmsm5c+dQKpWlc2McPHiQ7t27O7hUtzet1kBKeh5bVz3J\nvdM/ZeKozjz14gYKi3QM7d+SoXe14sdfTzC0v61tRKGQoVDIWLhoCzl5xchkZf+WKZVynp8zjJEP\nLOW7ZQ+xcNEWurRvUvoe2KZVAK479gontYT5/5nL+o1bbnibs23btipfo9VqAZOhtBEVEThu6LYJ\nJl988QVWqxW1Ws3bb7/Ns88+y969e4mKisJqtXLs2DFmzpxp13Ne2DmF3KSf8WpY/TlMy2WF/KpN\nkVIpiURK9qXDBHd7B79Wj9slT7nam0uxf9I44m6sVgtSYMOyKVhMhfy40jYfy+R7yy4UduW1Xmub\njHnVB/eVef/K/q/fG49em0f7Fp5cPLIQrMW8+NTV9WhemnsXem0eK5eMK/3/2uPB1ptjMesACVOn\nVv2JaQCFXA4mHRadAatRW61jBZvbJpgATJs2jddee40TJ06U7nvxxRcBWL9+vV3PZTGVkJv0M33H\nL0MqrXg4eFxCBg383HBxtjUWFhXrSrfLew2g1xtRqWzzixqNZhSK6g85Lym8h+O737NbMAkdvJmo\n9WGkJRxCKrfPPClWqxVJFSeZqoxUpqHFqL+QKao2olZiMSGxGJFajAzr0wlLcW69Xk2wvrutggnY\nJimePn166X2zWq3GarUSGxtLu3Y3P5gpLy8PJycn5FIpEqmi3ECyc985Bva52iq+4aejjBvekYgw\n2zoxl1LySrdffW8rJpMFlUrOs48PRamUU1Ssw6/NM5z682WaNfFl+tyvWbfsoWqXVSqzz5IPVyic\n/Ok0q8CueWZmZtbtCFirFanFhEKfB9f0TNkroN3Jbrtg0qhRI8aPH8+HH34IwDfffFN6m2MP7u7u\nLFmyhKfmPFbu+5FH41n7v8MM7NOCXX/F8MO2YyQkZzF2eIfS10/MvKs0vUqp4NS5RGZO6sW8lzay\n7J0pAAzu24Ln39rMxs9tUz4WFGp5+5PfmDauGzm5xezcH03Txj4cO5WMTmekY9tGtIloSI/O1ZtP\ntbpy9q+n6MwuLMbqLcZ1IyaTkWL5zc/wbrWYUAeG02DcC9cFBonFhNRiRGIx2rbNujKBpK6cPBPN\n2k1beHfZygrTpaRnoJDL+fSDJRWmS8/KxmKx8u2331aY7tKlS/z4448VpsnNzaWgoIAdO3ZUmK6g\noOCGwf+2CSavv/566UNfjz/+OFOm2L6UTk5OAPTp06fKeT377LNs3Ljxhu9brVZ+2PQdyx+/vjfh\nq/UHiIlPI79Ay+dr9vLd8od5+uWNSJCUvo6OTQNs3Z9R5y7Rt3tzki5lk3z56gjYBn7u9OjUlK/X\n/wXA91v/ZmDvCD78YifDBrTC3U1Dvx7N2X84jufnDOOFd37k1LnLtRpMsnZ8QeqmN2g27TW7LXRl\nN1YrKbvXkrL2vzSctgip2WALHmYDEkcOYrtGu1YRTBl7D8MGVrw28WervsPb04P77q14ANnaH35C\nbzDwyPyFFab78MMPmTdvXoVpdu7cSWRkZGmzwI0cPnyYH374odz3bptgcu2TpxKJBE9PzxrnVVRU\nxHPPPXfDKQU///xzZjwwhdTtHcvsLykxcDElh4G9W7B+yxEMBjOFRTqSL9mCxJXXV8QlZNCwgQfT\nJ/Rg4mOf075VcJn8ZkzsyaTHvkBvMKFRK0nLzMfDzTaFYuMgWy+Fr7crbi5qfL1dKS6p3XlQ0ra8\nS8unvsQluH4unu4e3o1jr4yg4YSFyCz2qTkJVSdam27AxcUFb2/v6/5JJBJmzZqFv7//dcckXc7m\nuSfu5oW5w5BJpSyYO4zFn/7OiEFt8PRwKn3t6WGrLXVo04gWYQG8s/Q3Pnh1Av6+tmc2lAo5A3qF\nA/DOC2MY0q8l40Z0IDevhOeeuJvQJn6ENvHDzUVNz87NUKsV9OsRxsDeEbX6mchdvFA419/Z4KWK\nf6YjMJscW5A71G1TM6krV8YtWExarBZTmd6IFmEBtAizPVD10NTeAHRq27j0WH9ftzKvAWZN7lW6\nfeVYpVLOxFG2dWiaBPswe7qtWjznQdu0jj7eVx9iGzHI9vzG2OFla0lX2PsJX0G4ERFMakgq19Bx\nZh7nT7yDRCIHO/QGaEtK0PzTxmMPcpWn3XtfBOFGRDC5CRKJBC+/MejT4sEO4xMsxiKcLPZ6dF6C\nU4MOlSerJW9/vIIH7hvNa+8tZcrYkfj5eLFjXyQASoWCzOxcSkpKeP2/83jzw884cOQYk0aPQCqV\n8sCE0ZyLjSf5UgoKuZwBfXo47DqEqhPB5CbkHfmJiyvn4NGmv926Gu1Vj7AY9SSveJRmz2/FqXFb\nO+VaNXEJSezY8xcSiYTw0Ka888nnLFv0MufjE3jm8Qf5at0PvPJ/c3hp0UfMmreAAH9f3nphPivX\nfk/HtraHAmMvJHLk+CkG9e1Zp2UXak4Ekxqy6Eu4uHIOred/i1ODpo4uTrmyT+7k8upnCVv4W52e\nNzSkMT27dOToiVMUl2hxcXbixOlo9hw4QmADf0q0Wv7YvZ/4pIvMnjGZDT/+woSHnuLh+ycyoHd3\ntFod7q6utAwPRamsZ13QNyE9M4v0zCyycvIqTHfo2ElcnJ3R6SvukYo8egyjyYx69eoK0x05coTV\nlaQ5c+YMcXFxlaaLj48nLS2t3PdEMKkpiRSpUlNvAwmAa5O2mIuW1vl5T5w+x+4Dh3hwyn18v/U3\n5HI5jYMb0rNLBywWCxNHjyAnN49uHdux/9Df5OTlM3PyONxdXZj48FwWv/Qs+w8f4/dde+naoS07\n90XSr2cX+nSz0zNQDmSxWDCbK35Y0Gq1VimdxWJLZ8iqeJlaU3FeFfKyYLVaq5yuPCKY1JQErDfZ\nBVlYVMwT/30VpVJB766d8PPxRiqVIJVKOXEmmoemjMfL8ya6Ymtx0fKKtG/dgkF9e9KmZThr/7eV\nDm1aEnUmGhdnJzIyszl64hRmi4XkS5dpFRHGxFHzePy5VwhvFsJv61fi5elB907tOR19nuISLTMm\njaNNJbOT3Qr8fX1oFR5a6aA1nV5fpUFrCoUcvcHAjEljK0yXX1hU6UOuVwatVZbujhi0dqtJy8hk\n3sK3aNsynLt6dmPF6vVMnzDa9iW7nEpi8iUUipsfZu4I8YnJSCRSOrVtxfQJozl3Pp6xwwdjNJlQ\nKhRMnziGH3/dwdmYOM7GxHEpJZ03nn+azu1sUzsmX07h/15dzIbPP6SouJhFn3yBTCalZfNQB1+Z\nUBExaM1BsnPzaBUexvFTZzl++izNmzXhjQ+Wc+hYFKnpmew7eJQPP1/FGx8sJy4hydHFrZZmTRrh\n4uzE82+8R+LFFAxGI82aNCL5UgoAryz5mDc/XE5gAz9CQxqza/9B/tx/kHumPcrLiz/G3dWVl+Y/\nwcJ3PiTpUgqTxoygedMmjr0ooVIimDhIRGhTvDzdWfifJ9j002+YzWZ+37CSx2dMRq/X07l9awoK\nC3lwynhCQxpXnmE9cyHpIn17dKZT25akZWQReyEJlUrFlLH38PxTj9Gzcweemf0gp6Njkcll/Lkv\nEp1eT+LFy5w9H8eR46fIzS/gyPFTvPD2B+QVFDr6koRKiNscB8nIyiE6Nh6DwcgX779RGjCefult\nnn50Bu1bt+BSShpvfbScd158Bmc7DmarC3KZjAZ+tqdLNWo1zZs1YWCfHhw8dpIdew5w+HgUb330\nGZ7ubvTt3qX0uMupafTo3IEenTvQrVM7WoQ141xsPBr1nbNk6q1KBBMHCfD35ZO3Xrpu/6pPFgG2\nVvOgwAblprkVFBYX88vOPQBcSrV1JX65diMrP3iLPt06U1RczKvPPsUbHyynpKSE5Mup9OjSgcKi\nIgCMRiMRobaeMp1Oj5NG45gLEapMBJM6ciYmloN/n6SouIR7hw7AYrGw/sdtvDBvNlt//xOAkUMH\nsPSrNVitVn7evpt7Bt+Fk0bDg1Nsa5Rs3PILE0YNd+RlVMu1nUkZWdl4ebiz+NMvKS4uIfLoCV5e\n/DE+3p7k5efTonkz4hOSiQhryqoNm9m2YzcL5j7GH3v+4sz5ODF50S1ABJM68tZHnzFp1Aj8vL14\nd9lKnnroAYbc1ZulX60hJu4CcQnJHD15GoCoczGMGjqQNZu2MHH0CPZEHmbb9t2cjo7l6MnTmExm\n3n/tJpdqqGVzH55O25a2J5+jzsbg7KTh6UdnEtjAtvrcq88+BdiWRC0qLuF0TCyNGgbQqGEger2B\ngX164OHuipNGw9OPTHfYddhTdSdH+mzVdxWmuzI50tpNP1WY7lJqOj/9VvGkR7n5BRQUFonJkW4F\nuXkFRJ2LAWzjCGQyKX/uP8iDU8ZzJjqWvQePMmPiGL5Ys5GeXTpQrC1hzIjBqFVK+vXoSr8eXVnw\n5nu89cJ8nnz+NQdfTeWuBJJrt8tr95HL5Xi4u9G7a6fSfSqVkqBA27SWzZtVbT7XW0FtTY40a/L1\nq+td66MvVjP34QcqTPPnvkgi/7bVFitS0TgT0ZtTR5o3bcIj90/kkfsn4uXhjlKpQKvV4unuRgM/\nXzw93Ahs4IeTkwalQsn5+ET8vL1JTL5MiVbLx1+uxs/Xh7z8AgqLih19OYJwHVEzqSN5BYWsWL0B\nALlMTkxcAv17defND5eXpnnzw+XMefB+Vm38kdCQxhQUFfHo9Els+W0nGrWah6dNYPazLzNr8jhH\nXYYg3JAIJjUmqda0A998/E65+/v1vH6B7nn/aiNoEtywdHv54lerfE4kEpDYsfJptSCpZFkPh5PK\nbC2/or22zonbnJqymJEq1OhzUhxdkhsqvhSNMa/8JzxrQu7hT150pN3ys7fiSzFYdEVI7BlAhSoT\nNZMakqqcCBi/kJNv34dPt1EOWTqhIlaLheyj2wh52n6LjwXdv4TY14eQe+4v5M41n7C7VlgtZB/9\nhUaPrgCVMxgKoZ7MSn+nEMHkJnj2vA+lXxP0qbFlpm08ceIE7du3r3Z+ZrPlhmvoVoXVem0xJIQN\nm4u6YXhFh1SLumE4EYuPUnRuP1bT1bk2kpKSaNy4ZkP+zWYzMln1b50KCwtwdXW7ukMiIXToHDTB\nLbECRrVn6Wp9UrMRrGKN4NomgslNcg7tgnPo1eHgVquVl59fxr4nF1U7rzVr1tCsWRN69Kj+NIW/\n//47Wq2W0aNHV/vY6lC4+eLZbUzp67S0NBZ9Mo/165+tdl4HDx4kJi6G6dOnVPvYj15+mREjRtC1\n6/VtTldYpQrMUgVmOWA1IzXbFuJyVINKbn4Bx0+dQy6v+Gt3LjYeNxcXPNzdKkx3+tx5jCYT2/f8\nVWG6mLiEStOcOHWWuIRk/vjjjwrTRUdHk5OTU+57IpjY2ZkzZxgxYgTHjh2jY8fyZ4wvT1xcHH37\n9kWhUJCRkYGfn1+Vj01NTaVt27ao1WoSEhIICQmpSdGrzWg0YrFYmDZtGrGxsYSFhVX52MzMTBo1\nakRAQEC1j71w4QKTJk0iMDCQ3Nzcqq2RJJFhkcsANSaFM0a1bZUBidWC1GIkLSuXdrW8zrCnuxut\nwkMZ3K9XheniE5Px9vSoNF1GVjZ6g6HSdGfPx1eaRiaVYjCZGNA6uMJ0LsZcLkZHlfueaKmys9at\nbXNyVCeQAISGhuLi4kJAQEC1AglAQEAAAQEBuLi41FkgAVAoFAQGBgJUKxgA+Pr6EhgYiLOzc7WP\nbdq0KTKZDHd395tabA3AKpFilqk4FBWDxNUPqYs3SMXf2JoQn5oglJIgkauQuflhtZjAZMBq1GE1\n6hGNuZUTwaQW/PDDD5w+fbraxxkMhpuaQPnrr7+u8bE3Y8GCBSxaVP02IrA9m1NZG0J5dDodR44c\nqdE5q0IilYNSjkRpewTAatJjNeqx6otq7Zy3OhFMakFliz/fSFFRES4uNV83pyZfSnuobEbziuTl\n5eHhUX+XHL1CIlchkatA7WwLKkYdVpPeYfPs1kcimNSCUaNG1ei4nJyc0uVHbyU16Qa/IisrCx8f\nHzuWppZJZEiUTldrLEb9P7UWHVju7DWORTARhJsgUaiQKFSgccNqMWPV5tvG4NSzQYx1QQQTQbAT\niVSGxNnLdutjNpbWWKxmo6OLVifu+GCiUqkwGo2o1eoy+0+fPs1vv9XtSnhFRUWMH1/x3BQ3y2Aw\noFJVbT5VhUKByVS7VffDhw/j7u5eq+f4t+joaEaMGFH62mw2I5VKazQSt1wSCciVSORKJGrX0t3P\nvPKObRGrSmaNe/g/LyCRSCpNN/nRpwEqTffkf1/BWskYmrvGNKPf6KmVpus8uBldh5b/1PodH0y8\nvLzIzMzE1fXqD71169bs2rWLCxcu1GlZDAZDrQaTgoICFApFlYOJl5cXWVlZtdqO89577zlkfaD5\n8+eXbmdlZdVJI7CjGsjrisR6o7X+7hAHDx4kJiaGsWMrXhWtLtR2A+wff/yBUqlk5MiRVUr/999/\nc/LkSe67775aK1N9aIDdunUrfn5+DBo0yKHluNXd8SNgO3XqREFBAf/73/8cXZRad+LECfr1q3jK\nwGu1a9cOrVZbiyVyvO3bt3Pu3Dl69uzp6KLc8u74mgmAVqvlrbfeIjg4mGHDhjls3ENt1UxKSko4\nduwYPXv2rHb7hE6n45tvvmHYsGG1UjZH1UwKCwv5+++/OXPmDHPmzMHZ+faZa9ZRRDD5h8FgYPfu\n3Rw7dgypVIpGo7nhau+1pbi42K6/1BKJBJ1Oh16vp2XLljW+ldu+fTvHjx/HYrGgVqvt+rnc7EC9\nmpBIJJSUlNC2bVv69etXpr1MqDkRTMqRn5/vkOp9lZ+ArQaVSoWbm5tdeiqKiors/rk4omYikUjw\n8vJCWstPCd9pRDCpR27VEbA3IzMz84brsAi3FhGaBUGwCxFMBEGwi9t7FE0dsFqtnD9/nv379xMb\nG0t2djadO3emR48etG7dWtyXC3eMG7aZpOfoWL4lkZxCw527aLTVSrswd2YNK3+y5Pz8fBYtWkRi\nYiL9+/enQ4cOeHt7s3//fnbu3Im3tzcvvfRSlbtjRZtJ5axWK+fOnSMyMpIzZ87g7e1N69at6d27\nN97e3jc8bv+pbE7E5mM033kP4AFYLODjrmDa4EbIZLXzfS43mBTrTIx+4RC9WzegWaAblju0idZs\nsbDnZAqdwt2ZM7Zpmfd0Oh1Tpkxh+PDhzJw587reEqvVyqeffsp///tfEhMTq9RjIYJJxdLS0nj3\n3XfJzMykX79+dOzYkezsbA4fPsyhQ4cYOHAgTz755HV//NbtuMjXvyYztHMQUumd+YdRJpUQeTad\n7AI9W9/ujlpp/8XUyg0mG3ddZn9UHvf1a2b3E95qDEYzH20+xQ+vdUUhv3rLMmTIECZPnszIkSNR\nKBS4ubmVW4P77LPP2LdvHytWrKh0PIUIJjeWnZ3NzJkzmTlzJqNHj77usy4oKGDRokXExcWxdu3a\n0udgtHoz9y44yLxxbXBzqvksdreL9btiGdjJm7F9A+2ed7k39EVaI/6e169Yb09HIndiNps5e+ow\nACXFhZyJOlTm3+WLVx+0i489TXraxTJ5mIxGUi4nUpBf/tT79qBUyLBYrBhMV6vHVquVQ4cO0a9f\nP3x8fHB3d2fdunVIJBLi4uLKHP/YY4/xySefMHXq1Bs+gfvtt9/y+eefI5FISE1NZdiwYfz555+1\ndk31walTp/jiiy9Yu3YtBw4cqDDtkiVLePrpp5HL5axcuZIRI0bw6qtll0l1c3PjzTff5LvvvmPW\nrFml+01mKxqVrFYDSVZmKnExUej1utJ98edPXff7fPb0EfQ6re3356/rl5RIuZxIWkpSrZUTIMDL\nmcKS2nkS3CENsAaDnh/WLyM5MYaUy4mcO32UpqGt+Xnz12RmXMbdwxtXN080Ti706T+SP//YhK9v\nIGaLmbzcLLr2GMR3qz9kyPBJAGRnpTHjkQV1Vv4zZ84QHR1NQEBA6b6pU6fi5OTEsmXLeP/998uk\n9/LyonHjxpw5c4Z27dqVeS86OpqTJ0/y7rvvlu7bsmULH3/8Ma1bt672TPW3giNHjrBu3TrmzJmD\nRqNh/vz5FBQUcPfdd1+XNi8vj9WrV6NUKtm6dSuBgYFotVoeeOABtFotGo2mTHqpVMqjjz5aV5cC\nwKZ1n+Lp5cvJY/sBaNOhJ9+uXIxBryMgsDEX4s4glckICGxMSGgrTCYjRYX5HI7cQffeQ9m0dinB\nTZoTFt6W/Xu28fqSdXVafntxSDDZ8esGGgY3Q6V24oGHnuO3rWswmgyERbSjaVgrYmOiCA1vS0lx\nIb/+9C06bQlBwbZbLoNBj69/Q5qHt2PAkPHk52Vz/OjeOi3/0aNHmTFjxnX77733XlasWFHuMe7u\n7ixfvpzmzZuX2Z+UlMTYsWOJjCy7hm+LFi1ISEi47YJJVlYWy5cvZ+nSpTg52Wq/a9eu5ZNPPiEk\nJITw8LIrEG7evJmCggLi4uJKpyrQaDR8//33fPnllzz00EPXnaNbt27odLrr5qipDVptMdt/Wc97\ny7aSlZlKQvxZXFzcaR7eDmdXdw799Qd3DR5L8T/BIy01mS49BuHh4U1uTiY+vgFIZTImTJuDn38Q\nSYnna73MtcUhwSQvNxOVWoPJZCQ97SJOzq507TGITeuW4uzijtls4kjkDoaPmk7T0FZs3riCqOMH\nkMnkOLu4sWndpzi7uPHD+mWUFBdxKTmOEaOn11n5//0Lf4VMJiv9gvzbgQMH0Ov1172fnZ0N2O75\nryWRSLBYbr+eh8TEREaPHs3582W/NO3atWPVqlXXraGzZs0a+vTpU+6cJ0ePHi03mMjlctLT02nY\nsKF9C1+O6DN/07hpOP6Bjdi6+WuKCvO4b+qTKFVqTp88iFKpKv0/uHEonl7+pKckk5+Xjdli5u9D\nu8lMv8zXn72Bf0AjLiae5+Sx/bTr2LvWy25vDgkmU2bMZ+Wy1+h91z1s3rCC+6Y+CYBKpaFJ0wgA\n8nKzMJtNKJUqLGYzRUX5yGVyJFIp4yY9RmZGCn/t3kZwkzDc3Ot2Ee2KGgz/XfMA23IOycnJfPTR\nRwwfPrzMe6dPn+bw4cNl7vOtVivz58/nmWeesV+h64krS1skJZVtG/Dw8CA6OprU1NQy+w0GA337\n9i03r5KSkhueJyEhoU6CSYfOfTl2eDcajTPakiIah9j+0BgMeho1aV7aUJyeepFmzdtwJuoQCrmC\n4uICzGYTXj7+9Oo3AqlMhtVqpWFwM4wGQ0WnrLccNmitID+HP3//HqlMxh/b1hHRqjN5edmkpyYD\nUG6mUP8AAA9HSURBVFiYT3JiDB6efnTtOZic7AyUKjUGvQ4nZzf+2v0FoeFtaRDYiOQ6rhqGhoaS\nmppaps3kinnz5rFr1y769+8P2J4EbtiwIYGBgQwZMuS69C1btuTAgQM8++yzvPHGGxiNRt5++216\n9OhRulre7SQ4OJijR4/yxBNPlOmRef/991m4cCEdOnQok/7nn38mOjq63LwyMjLKXQ41LS2t3KBe\nm/78YxM+foFkZ6axf9dW0lKSsFjMpdeYm5NJRtol3Nw8CQwKIaCkiNTLiTg5uZCWkoRer2Py9Kf5\n3/rl+PrXfhCsDeV2DX/1SyKpWVb6tr3+y2IvMeeO0zDo6tiN/LxsdLrr/9I0bhJOUWEeMoUCmUxO\nWkoSDYObYTDokEllJCXEENSoGa5utVc7WbT+OBte6YKz+mrsfeutt1iwoPxG30mTJqFSqXB3d+fE\niRN4e3tz1113MXfu3Bue4/vvv+f8+fPI5XI6dep0W8/6tWnTJnbv3l06ZeM333xDQkICr/5/e/ce\nFWW5L3D8y224yFVkVEQFhRBBdpqQB1G2d5flqnboSttesmPHZeHuaJq2RNlKrDBb7pVnYakLLRaE\n5TFR0UrJc1JTRCXoDFiD5OaSiFAMzDgMw8x7/uDIycRLwwsvyvP5x7XkfZ/5/dYafjzv8z6Xv//9\njhnDtbW1LFu2rMPNq06fPs2ECROora1t7y3W19cTEBDQvrq56WYrLyZfYPVc24/juB/tD0UEDAnG\n8n9v61pamvn1lxt3XHfru2wwNDFwUCCXNRcIi4jil/rruLi4oW9qwGRqZkhg1xXC/y76mcH9HVk0\nc4jsbXfYM5EkunxyT2jY7X+B3D3uPkvUu+//P1YMCw4HwNm5bXAtLGJsF0R3Owd7uztOh9RoNGRk\nZLBgwYI7rs/MzKSqqgqj0Uh1dTUrVqxg9+7d9/yMOXPmUF9fj5eX1yO/V2h8fDx2dna88847ODs7\nExQUxIYNGzpceqBWqwkMDCQ3N/e2TaABYmNj2b9/Py+//DKPPdb2SJGXl9dhD7ArhYT+6Xf/40Vf\n3/4dXuvd16/9+zxyVDQAvv0GANDH3bPLYuwO95wBO6hfH6JD/XrtDNhWi5Xc/AoWTB/Mwhl3VnKd\nTsfatWuxt7dn6tSpPP744+3T6U+cOEFDQwMbN25k6NCOp+P/Vn19PVu2bMHJyYmkpKRHvqAAfPrp\np5SUlJCUlHTfa8+cOUNsbCxFRUVERkZ2eE1GRgYZGRkcOnTotjc572Zr0VYaifuTf9sfhl7I3s4O\nbbWO/yr6mc+Tn8TdVf7v113X5phbraQf/Sd1ut67NkflaMeYx7yZPObuA64Wi4Vz585x6tQpysrK\n2hf6xcTEMGHChAcuCrW1tTg6OuLl5UVDQ8M915k8CiRJYs2aNRgMBlJSUh5oq8zvvvuO5ORknnzy\nyfa1UHV1dVy4cIEjR45gb29PUlLSHQPkkiTxj/1XKP2nnl5aS5Ak6Oel4rW/DGOgb9e8MhebI/Ug\nvWk6fV1dHR4eHuh0Osxm8wO/edHr9Rw5coRz585RUlJC3759CQ8PZ/LkyYwbN06+s2+EP0wUkx6k\nNxWTW8ROa4+OR//BvIuZG+swXiu7/4UP4GZjI4435BmEs7N3wG1oBA4q1/tf/AdYjE0YKkuQLPKs\n77jZ0ICuTobTACQJ10GhqLxEYVKK6Jl0gv6nIorXT8IjJLrtSMgexNxYR3PNFZ54/3ucfeWZt9Dy\naw0XXhuJ+7DR2Ku6fqr6H2FtbUVfVkDk5jzch42+/w2C7EQxsZG1pZlzL/kT+u8Z+I596v43KKDy\nP1Np+P4ko5LkOTP52xf74j/rVQJf3CxLe3JrKP4a7YevMWbbRdl7ZML9icecTrBzVNlcSKqqqsjK\nyiI4OJji4mKgbfwgMDDwtnU6ISEhLFy40KbPUP/5r1w/+bFN93bEuV8AA6Yvtene7sjXO3IyFmMj\nWC023S90jigmCgkICCAyMpKrV6+2L2IzGAysXr1a4ci6Rm/LtzcSxUQheXl5HD9+nMWLF3Pp0iV8\nfX1pbm4mMTGR/Px8wsLCqK6uJiwsjMWLFzN8+MO9611vy7c3ElunK2TKlCm4uLhw7Ngxxo0bx5kz\nZ7Czs2PDhg2Eh4eTkJDApEmT2LRp0yPxi9Xb8u2NRM9EQZWVlVRUVGA0GvH09MRoNJKWlsb48eOB\nttm1hw8fRqVSdbgL2cOmt+Xb24ieiUJaW1tRq9WkpaWxceNG+vXrh729PW5ubsTHxzN8+PD2Yx0m\nT56sdLid1tvy7Y3Eq2EbWVuayV8ayL98VNMl7Vsslk5PDTfVV/P9xumM/Q+NLDFd/Fsk4euP4OIn\n//J1OfIFOLckgKi0yzi43PskAEF+4jGnm6xbtw5JknB3d2/fRBnatjLYtm1b+xYFmzZtAuCrr75i\n+vTpeHh4sGLFioduzcmtfGtqahg1atQjn68gikm3+PbbbyksLMTT0xOTyYS/vz+lpaWEhYWRm5vL\nzZs3KSwsZPTo0ahUbUcyODg4oFKpMJvNZGdn88MPP9zW5ty5c4mIiFAinfv6bb5ms5mysjKb8j1/\n/jzR0W17fvTkfIU2YsykG8TExODh4YGbmxszZsxg4sSJhIaG0tTURGlpKenp6WzduhVJkpg2bRon\nTpwgLCyMkJAQampqmDNnDmazmYSEBGJiYoiIiOjRv1i/zdfDw8PmfENDQx+KfIU2oph0k8GDBxMS\nEsITTzxBbm4ujY2NLFiwgNWrV+Pi4sILL7zA8uXLSUlJITIykiVLlrB9+3bGjBlDQUEB3t7e+Pn5\nMXPmTL7++mvKy8vv/6EKupUvYHO+JpPpoclXEMWk2zg7OxMcHMzHH3/MwoUL2bdvH2+//TYjR44E\nYPbs2cyYMYP9+/dz5coVLl68yLFjx7BarTg4OLBq1SoSEhKAts2XhwyRfxBUTrfy3bZtm835xsXF\nAQ9HvoJ4m2Mza0sz5/9tOOP2VCsdyl21/HqN4g3TGLv9f2Rp7+LfIonYcBRn3wBZ2usK+f8ayNjt\n3+Pg6qF0KL2O6JnYSJKs2Ktcab5RoXQod6W/UkjLLz/L1p6jhy/68u9ka09uzTcq2xb62YmvtRJE\nz6STdJpTNJUVAJ3fz8RsNnd4cp0t7Bwc8AqfiHuQvEc8GK+V8WvhV0gWsyzttbS0tL/R6QwJCfeg\nx/GO+HPngxJsIopJD7Jt2zaCg4OZPXu20qF0i9zcXDQaDWvWrFE6FEEGoj/YQxiNRoqKijh+/LjS\noXSb48ePo9FoMBgMSociyED0THqIq1ev4ubmhqurK0ajEbVarXRIXerGjRu4uLhgMBgwmUwPdLaQ\n0LOJYtKDiN3phYeZeMwRBEEWopgIgiALUUwEQZCFKCaCIMhCFBNBEGQhiokgCLIQxaQHcXfvfVsN\nenrKc7ayoDwxz0QQBFmInokgCLIQxUQQBFmIYqKAiooKZs+ezalTp7hw4QJTp05lxowZ7Qd679+/\nn5UrVyocpbzee+89oO184ZdeeomsrCwA9Ho9AFu2bMFisWA2mzGZTIrFKdhOFBMF6PV6TCYTubm5\nxMfHM2zYMNRqNfHx8QDEx8ej0Wi4ePGiwpHKY9euXRw4cICkpCSmTZvGrFmzyMjIYOvWrbzyyisA\nFBYWYrVa+eabb/joo48UjliwhTjqQgFVVVX4+voyc+ZM9Ho9arWap556irNnzwJQU1OD1WolOTmZ\nzz//XOFoO8/Hx4eRI0cyYMAApk+fTklJCbNmzaK0tJSmpia0Wm37v1VVVUqHK9hIFBMFODg4oNPp\nOHr0KKWlpVRUVNDS0tLevd+xYweJiYkUFhaSlZXF/PnzFY64c/Ly8tBqtQwfPpzKykqcnJxwc3Oj\nT58+xMXFkZOTQ3V1NYcPH6axsZEJEyYoHbJgA/GYowBfX1/8/PzQ6/X8/s38oUOHKC4uZuLEiSxf\nvpydO3ei1WoVilQeKSkphIeHExQUBIAkSeh0OgDc3Nx44403GDFiBCtXrmTdunWcPHlSyXAFG4li\nogC1Ws2YMWNYunQp0dHRBAcHM2/ePBoaGkhLSyMzMxMAJycndu/eTXx8PD/++KPCUdsuOzub/Px8\niouLMZvNSJLUfuTFrQHYW6xWKw0NDUqEKXSSmLSmkOLiYvz8/MjJySEsLAyz2YxKpSIqKgpXV9fb\nrr1+/TpqtRo7u85vWq2UtWvXMmfOHPbu3UtQUBAWiwVoO3vY2dkZo9GIi4sLra2t+Pn5kZqaqnDE\nwh8mCYKMCgoKJKPRKKWnp0uSJEmZmZlSbW1t+8+rqqqk1NRUqa6uTtq6davU3Nx82/3r16+Xmpqa\nJEmSpOrqaunQoUPSrl27pGvXrklz586VTp8+LZ0+fVoqLS3tvqSEByIGYAXZ7Nu3D4PBwNixYzl/\n/jxDhw5l+/bt5OXlMXjwYJKSkkhOTub5559HpVJRUlLCu+++y9SpU9mzZw86nY7CwkJqa2sBeO65\n58jJyWHSpEm8+uqrDB06lKKiInbt2kVGRobC2Qq/J4qJIJvMzExycnLQaDRMmjSJuro6pkyZQkJC\nAv379+fq1avs3r2b8ePHk5iYyMmTJ3FwcGDQoEHs2LGDp59+mmPHjjFs2DAAvvzyS0pLSzEYDCxZ\nsoT09HRKSkr45JNPGDFiBEaj8Y5HQkE5opgIsjCbzTg7O2NnZ4ePjw/Z2dk888wzZGVlodPpcHR0\nxMnJidjYWKKionj22WcpLy9n586dAGzatAmNRnPbWMkHH3zAqlWrCAoKYt++fXzxxRckJiai0Wjo\n168flZWVjB49WqmUhd8RxUSQnb+/Pz4+PixatIgDBw5QW1vL+++/j4+PD2+++Sa+vr64u7ujVqux\nWCycPXuWgoIC4uLi+PDDD9vbOXjwII6Ojri7u+Pq6sqyZcuIjo7mrbfeIiYm5o7X6oKyRDERZOHk\n5ITJZEKSJMxmM+Xl5WRlZXHw4EG0Wi3r1q0jPT39tnsWL15Meno63t7eZGZmMm/ePPLz89t/rlKp\nWLJkCVFRUVy6dAmtVktqaiopKSkMHDgQLy+v7k5TuAdRTATZzJ8/n7179+Lo6Mjrr79OUVERK1as\noLW1FScnpzvmysTGxrJ582Y+++wzPD09aW5u5vLly+0/X7RoEWlpaZjNZs6cOcOQIUPYs2cP/v7+\nlJSUtM9VEXoGMc9EEARZiBmwgiDIQhQTQRBk4fjTTz8pHYMgCA+5Rp2O/wUsqgYgG2sJfwAAAABJ\nRU5ErkJggg==\n"
    }
   },
   "cell_type": "markdown",
   "id": "5ab892ff",
   "metadata": {},
   "source": [
    "## 注意力层\n",
    "\n",
    "自注意力（Self-Attention）操作是基于Transformer 的机器翻译模型的基本操作，在源语言的编码和目标语言的生成中频繁地被使用以建模源语言、目标语言任意两个单词之间的依赖关系。\n",
    "将由单词语义嵌入及其位置编码叠加得到的输入表示为{X<sub>i</sub> ∈ R<sup>d</sup>}<sup>t</sup><sub>i=1</sub>，为了实现对上下文语义依赖的建模，引入自注意力机制涉及的三个元素：查询q<sub>i</sub>（Query）、键k<sub>i</sub>（Key）和值v<sub>i</sub>（Value）\n",
    "\n",
    "![transformer结构图.png](attachment:caff02a6-bcee-4ccd-9247-abbb111f0ca0.png)\n",
    "\n",
    "为了得到编码单词 x<sub>i</sub> 时所需要关注的上下文信息，通过位置i 查询向量与其他位置的键向量做点积得到匹配分数 q<sub>i</sub>·k<sub>1</sub>,q<sub>i</sub>·k<sub>2</sub>,...,q<sub>i</sub>·kq<sub>t</sub>\n",
    "为了防止过大的匹配分数在后续Softmax 计算过程中导致的梯度爆炸及收敛效率差的问题，这些得分会除以放缩因子 $\\sqrt{d}$     以稳定优化\n",
    "放缩后的得分经过Softmax 归一化为概率，与其他位置的值向量相乘来聚合希望关注的上下文信息，并最小化不相关信息的干扰。上述计算过程可以被形式化地表述如下：\n",
    "        Z = Attention(Q,K,V) = Softmax($\\frac{QK^T}{\\sqrt{d}}$)V\n",
    "\n",
    "为了进一步增强自注意力机制聚合上下文信息的能力，提出了多头自注意力（Multi-head Attention）机制，以关注上下文的不同侧面。\n",
    "具体来说，上下文中每一个单词的表示 x<sub>i</sub> 经过多组线性{W<sub>j</sub><sup>Q</sup>,W<sub>j</sub><sup>K</sup>,W<sub>j</sub><sup>V</sup>} <sup>N</sup><sub>j=1</sub> 映射到不同的表示子空间中\n",
    "\n",
    "线性变换W<sup>O</sup> ∈ R<sup>(Nd<sub>v</sub>)</sup> 于综合不同子空间中的上下文表示并形成自注意力层最终的输出$\\{x_i ∈ R^d\\}^{t}_{i=1}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7483c39",
   "metadata": {},
   "source": [
    "# AI简单易懂的解释自注意机制和多头自注意力机制\n",
    "## 自注意力机制（Self - Attention Mechanism）\n",
    "\n",
    "### 基本概念：\n",
    "\n",
    "自注意力机制就像是一个能让信息自己和自己互动的工具。想象你在阅读一篇文章，你会根据每个句子和其他句子之间的关联来理解全文。自注意力机制也是这样，它会去看输入数据（比如一句话中的每个单词）之间的相互关系。\n",
    "\n",
    "### 工作过程：\n",
    "\n",
    "首先，对于输入的一组数据（例如一个句子中的单词向量序列），它会为每个数据元素（单词）计算三个向量，分别是查询向量（Query）、键向量（Key）和值向量（Value）。这就好比对于每个单词，都有三个不同的标签，用来帮助衡量它和其他单词之间的关系。\n",
    "\n",
    "然后，通过计算查询向量和键向量的相似度（这个相似度可以用点积等方法来衡量），得到一个分数。这个分数就表示这个单词和其他单词的关联程度。比如，如果一个单词和另一个单词的关联分数很高，就说明它们在语义等方面很紧密。\n",
    "\n",
    "接着，把这些分数通过一个 softmax 函数进行处理，让它们变成概率分布的形式。这样就可以根据这些概率来分配每个单词对应的 “注意力权重”。\n",
    "最后，用这些注意力权重和值向量进行加权求和，得到每个单词经过自注意力机制后的新表示。这个新表示综合考虑了这个单词和其他单词的关系，让每个单词的表示更加丰富。\n",
    "### 应用场景示例：\n",
    "\n",
    "在机器翻译中，自注意力机制可以帮助模型更好地理解源语言句子中每个单词的重要性以及它们之间的关系，从而更准确地翻译出目标语言句子。例如，在翻译 “我喜欢红色的苹果” 时，它能通过自注意力机制发现 “红色的” 和 “苹果” 之间的紧密联系，从而更合理地翻译这个短语组合。\n",
    "\n",
    "## 多头自注意力机制（Multi - Head Self - Attention Mechanism）\n",
    "### 基本概念：\n",
    "\n",
    "多头自注意力机制就像是多个自注意力机制同时工作。可以把它想象成有几个不同的 “专家”（每个头就是一个专家），每个专家都用自己的方式来审视输入数据之间的关系。\n",
    "\n",
    "### 工作过程：\n",
    "\n",
    "它会并行地运行多个自注意力机制。比如，对于一个输入序列，我们设定多头的数量为 h（通常是 8 或者 16 等），就会有 h 组不同的查询、键和值向量的线性变换。\n",
    "\n",
    "每个头都会按照自注意力机制的步骤（计算相似度、得到权重、加权求和）来生成自己的输出表示。这就好像每个专家都根据自己的判断，给出了一个关于输入数据关系的结论。\n",
    "\n",
    "最后，把这些头的输出表示拼接起来或者通过其他方式（比如线性组合）合并成一个最终的输出。这样就综合了多个 “专家” 的意见，让模型能够从多个角度去理解输入数据之间的关系。\n",
    "\n",
    "### 应用场景示例：\n",
    "\n",
    "在预训练语言模型如 BERT 中，多头自注意力机制发挥了巨大的作用。因为语言是非常复杂的，不同的语义、语法关系可能需要从不同的角度去捕捉。多头自注意力机制可以同时捕捉句子中单词之间的多种关系，比如语法结构关系（主谓宾等）和语义关联（近义词、上下位词等），从而生成高质量的词向量表示，用于后续的文本分类、问答系统等多种自然语言处理任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74fc8f70",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, heads, d_model, dropout = 0.1):\n",
    "        super().__init__()\n",
    "    \n",
    "        self.d_model = d_model \n",
    "        self.d_k=d_model // heads\n",
    "        self.h = heads \n",
    "        \n",
    "        self.q_linear = nn.Linear(d_model, d_model)\n",
    "        self.v_linear = nn.Linear(d_model, d_model)\n",
    "        self.k_linear = nn.Linear(d_model, d_model)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.out = nn.Linear(d_model, d_model)\n",
    "        \n",
    "    def attention(q,k,v,d_k, mask=None, dropout=None):\n",
    "        scores =torch.matmul(q,k.transpose(-2,-1)) / math.sqrt(d_k)\n",
    "        \n",
    "        #掩盖那些为了填补长度而增加的单元，使其通过Softmax计算后为0\n",
    "        if mask is not None:\n",
    "            mask = mask.unsqueeze(1)\n",
    "            scores = scores.masked_fill(mask == 0,-1e9)\n",
    "        \n",
    "        scores = F.softmax(scores,dim = -1)\n",
    "        \n",
    "        if dropout is not None:\n",
    "            scores = dropout(scores)\n",
    "            \n",
    "        output = torch.matmul(scores,v)\n",
    "        return output\n",
    "    \n",
    "    def forward(self,q,k,v, mask=None):\n",
    "        bs = q.size(0)\n",
    "        #利用线性计算划分成h个头\n",
    "        k = self.k_linear(k).view(bs,-l, self.h, self.d_k)\n",
    "        q = self.q_linear(q).view(bs,-l, self.h, self.d_k)\n",
    "        v = self.v_linear(v).view(bs,-l, self.h, self.d_k)\n",
    "        \n",
    "        # 矩阵转置\n",
    "        k = k.transpose(1,2)\n",
    "        q = q.transpose(1,2)\n",
    "        v = v.transpose(1,2)\n",
    "        \n",
    "        #计算attention\n",
    "        scores = attention(q,k,v,self.d_k, mask, self.dropout)\n",
    "        \n",
    "        #连接多个头并输入最后的战性层\n",
    "        concat = scores.transpose(1,2).contiguous().view(bs, -1, self.d_model)\n",
    "        \n",
    "        output = self.out(concat)\n",
    "        \n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cf56bae",
   "metadata": {},
   "source": [
    "# 前馈层\n",
    "\n",
    "前馈层（Feed-Forward Network, FFN）接收自注意力子层的输出作为输入，并通过一个带有 ReLU 激活函数的两层全连接网络对输入进行更复杂的非线性变换。实验证明，这一非线性变换会对模型最终的性能产生重要的影响。\n",
    "\n",
    "$FFN(X) = Relu(xW_1 + b_1)W_2+b_2$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5c1b36d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeedForward(nn.Module):\n",
    "    def __init__(self, d_model, d_ff=2048,dropout =0.1):\n",
    "        super().__init__()\n",
    "        # d_ff默认设置为2048\n",
    "        self.linear_1 = nn.Linear(d_model,d_ff)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.linear_2 = nn.Linear(d_ff,d_model)\n",
    "    def forward(self,x):\n",
    "        x = self.dropout(F.relu(self.linear_1(x)))\n",
    "        x = self.linear_2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "179d8feb",
   "metadata": {},
   "source": [
    "# 前馈层的简易理解\n",
    "\n",
    "### 一、整体作用\n",
    "\n",
    "前馈层的主要作用是对输入的数据进行更深入的处理。想象一下，数据经过了自注意力子层后，有了一定的 “初加工”，但是还不够，前馈层就是要把这个 “初加工” 的数据变得更有用。\n",
    "\n",
    "### 二、具体操作\n",
    "\n",
    "#### 输入来源\n",
    "\n",
    "它接收自注意力子层输出的数据作为自己的输入。就好像自注意力子层把 “食材” 准备好了，送到前馈层这个 “厨师” 这里。\n",
    "\n",
    "#### 网络结构\n",
    "\n",
    "前馈层是一个两层的全连接网络。这就好比有两个处理步骤。全连接意味着每一个输入的元素都和下一层的每一个神经元相连接。\n",
    "\n",
    "#### 激活函数\n",
    "\n",
    "两层全连接网络中间有一个 ReLU 激活函数。激活函数就像一个 “筛选器” 或者 “规则制定者”。ReLU 函数会让大于 0 的值保持不变，小于等于 0 的值变成 0。这样做的目的是给数据添加非线性因素。因为如果没有激活函数，多层的全连接网络就相当于一个大的线性函数，而实际的数据关系往往是非线性的。通过 ReLU，数据可以被 “塑造” 成更符合实际情况的样子，进行更复杂的非线性变换。\n",
    "总的来说，前馈层就像是数据处理流水线上的一个重要环节，它把经过自注意力子层处理的数据进行进一步的、更复杂的加工，让数据变得更有价值，更能符合模型最终的处理需求"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "141d8fd4",
   "metadata": {},
   "source": [
    "# 全连接的简易理解\n",
    "\n",
    "### 一、基本概念\n",
    "\n",
    "#### 连接方式\n",
    "\n",
    "全连接层中的每个神经元都与前一层的所有神经元相连接。想象你有一张由很多小方格组成的网格（代表前一层的神经元），全连接层的每个神经元就像一根绳子，这根绳子的一端系在新的一个点（代表全连接层的神经元）上，而另一端则分散开来，系在前一层的每一个小方格上。这就是所谓的 “全连接”。\n",
    "\n",
    "#### 数据处理\n",
    "\n",
    "全连接层的神经元会对它接收到的所有输入进行加权求和。假设前一层有 n 个神经元，输入的数据是 x1,x2,...,xn，全连接层的一个神经元有对应的权重 $w_1,w_2,...,w_n$。这个神经元接收到的数据就是 $w_1x_1 + w_2x_2+...+w_nx_n$，然后通常还会加上一个偏置项 b，得到 $z=w_1x_1 + w_2x_2+...+w_nx_n + b$。\n",
    "激活函数\n",
    "\n",
    "得到加权求和的结果 z 后，一般会通过一个激活函数（如 Sigmoid、ReLU 等）进行处理，得到这个神经元的最终输出。激活函数就像是给计算结果 “定个型”，决定这个神经元最终输出的值的范围和特性。\n",
    "\n",
    "### 二、作用\n",
    "\n",
    "#### 特征整合\n",
    "\n",
    "全连接层可以将前一层提取到的局部特征进行整合，得到更全局化的特征表示。例如在图像识别中，前面的卷积层可能提取了图像的局部边缘、纹理等特征，全连接层则可以把这些局部特征综合起来，判断出整个图像是猫还是狗等更高级的类别。\n",
    "\n",
    "#### 分类决策\n",
    "\n",
    "在很多神经网络模型中，全连接层常常用于最后的分类阶段。它把经过多层处理和特征提取的数据进行综合判断，输出不同类别的概率等信息，从而实现分类的目的。比如在手写数字识别中，全连接层会根据前面层提取的手写数字的特征，来决定这个数字最有可能是 0 - 9 中的哪一个。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "032cea90",
   "metadata": {},
   "source": [
    "# 残差连接与层归一化\n",
    "\n",
    "由Transformer 结构组成的网络结构通常都非常庞大。编码器和解码器均由很多层基本的Transformer 块组成，每一层中都包含复杂的非线性映射，这就导致模型的训练比较困难。\n",
    "因此，研究人员在Transformer 块中进一步引入了残差连接与层归一化技术，以进一步提升训练的稳定性。具体来说，残差连接主要是指使用一条直连通道直接将对应子层的输入连接到输出，避免在优化过程中因网络过深而产生潜在的梯度消失问题：\n",
    "\n",
    "$x^{l+1} = f(x^l)+x^l$\n",
    "\n",
    "其中 $x^l$ 表示第 l 层的输入，f(.) 表示一个映射函数。此外，为了使每一层的输入/输出稳定在一个合理的范围内，层归一化技术被进一步引入每个Transformer 块中：\n",
    "\n",
    "$LN(x) = α·\\frac{x-μ}{δ} + b$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb5a9c80",
   "metadata": {},
   "source": [
    "# 残差连接与层归一化简易理解\n",
    "\n",
    "### 残差连接\n",
    "\n",
    "#### 基本概念\n",
    "\n",
    "想象你在爬楼梯，正常情况下你是一步一步往上爬。但残差连接就像是给你装了一个 “电梯”，如果你爬得很费劲或者感觉爬不上去了，你可以直接坐电梯上去。在神经网络里，这个 “电梯” 就是残差连接。\n",
    "\n",
    "具体来说，在一个神经网络层中，假设输入是x，经过这个层的正常变换得到y=F(x)（F代表该层的操作，比如卷积等）。有了残差连接后，实际的输出就变成y=x + F(x)了。这里的x就是那个 “电梯”，它直接把输入加到了经过变换后的结果上。\n",
    "\n",
    "#### 作用\n",
    "\n",
    "解决梯度消失问题：在深层神经网络中，当反向传播时，梯度可能会越来越小（梯度消失），导致网络难以训练。残差连接可以让梯度更直接地回传。因为有了x这一项，梯度可以通过这条 “捷径” 更顺畅地流动，使网络更容易训练。\n",
    "\n",
    "提升训练效果：允许网络学习到恒等映射（即F(x) = 0时，y=x），这样网络可以更灵活地选择是对输入进行变换还是保持不变，有助于提高模型的准确性和收敛速度。\n",
    "\n",
    "### 层归一化\n",
    "\n",
    "#### 基本概念\n",
    "\n",
    "可以把层归一化想象成给一群学生 “调整身高”。假设一群学生（代表神经网络中的神经元）身高参差不齐（代表神经元的激活值有大有小），层归一化就是要把他们的身高都调整到一个比较合理的范围。\n",
    "在神经网络中，对于某一层的所有神经元的输入（假设为$x_1,x_2,...x_n$），层归一化会先计算它们的均值μ和方差$σ^2$，然后对每个输入$x_i$进行如下变换：y_i = $\\frac{x_i - μ}{\\sqrt{σ^2 + ε}}*γ + β$，其中ε是一个很小的数防止除零，和是可学习的参数。\n",
    "\n",
    "#### 作用\n",
    "\n",
    "加速训练：在训练过程中，神经网络的输入分布可能会不断变化（内部协变量偏移问题），层归一化可以使每层的输入分布保持在一个相对稳定的状态，这样梯度下降算法就能更快地收敛，加速训练过程。\n",
    "\n",
    "提高模型泛化能力：通过稳定神经元的激活值分布，层归一化可以让模型对不同的输入数据更加鲁棒，不容易过拟合，从而提高模型的泛化能力。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb96e6ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Norm(nn.Module):\n",
    "    def __init__(self,d_model,eps = 1e-6):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        self.size = d_model\n",
    "        \n",
    "        #层归一化包含两个可以学习的参数\n",
    "        self.alpha = nn.Parameter(torch.ones(self.size))\n",
    "        self.bias = nn.Parameter(torch.zeros(self.size))\n",
    "        \n",
    "        self.eps = eps\n",
    "        \n",
    "    def forward(self,x):\n",
    "        norm=self.alpha * (x - x.mean(dim=-1,keepdim=True))\\\n",
    "        / (x.std(dim=-1,keepdim=True) + self.eps) + self.bias\n",
    "        return norm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cd54cc7",
   "metadata": {},
   "source": [
    "# 为何解码器要比编码器多一个注意力掩码，使用掩码多头注意力（Masked Multi-Head Attention）\n",
    "\n",
    "主要是因为在翻译的过程中，\n",
    "\n",
    "编码器端主要用于编码源语言序列的信息，而这个序列是完全已知的，因而编码器仅需要考虑如何融合上下文语义信息。\n",
    "\n",
    "解码器端则负责生成目标语言序列，这一生成过程是自回归的，即对于每一个单词的生成过程，仅有当前单词之前的目标语言序列是可以被观测的。\n",
    "\n",
    "这一额外增加的掩码是用来掩盖后续的文本信息的，以防模型在训练阶段直接看到后续的文本序列，进而无法得到有效的训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a61719d",
   "metadata": {},
   "source": [
    "# 解码器另一个额外模块——多头交叉注意力\n",
    "\n",
    "解码器端额外增加了一个多头交叉注意力（Multi-Head Cross-Attention）模块，使用交叉注意力（Cross-Attention）方法，同时接收来自编码器端的输出和当前Transformer 块的前一个掩码注意力层的输出。\n",
    "\n",
    "多头交叉注意力（Multi-Head Cross-Attention）模块中查询是通过解码器前一层的输出进行投影的，而键和值是使用编码器的输出进行投影的。\n",
    "其作用是在翻译的过程中，为了生成合理的目标语言序列，观测待翻译的源语言序列是什么。\n",
    "\n",
    "基于上述编码器和解码器结构，待翻译的源语言文本，经过编码器端的每个Transformer 块对其上下文语义进行层层抽象，最终输出每一个源语言单词上下文相关的表示。\n",
    "\n",
    "解码器端以自回归的方式生成目标语言文本，即在每个时间步t，根据编码器端输出的源语言文本表示，以及前t − 1 个时刻生成的目标语言文本，生成当前时刻的目标语言单词。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f3fdbb9",
   "metadata": {},
   "source": [
    "# 简单理解多头交叉注意力\n",
    "\n",
    "1. 注意力机制基础\n",
    "\n",
    "在深入多头交叉注意力模块之前，先简单回顾一下注意力机制。注意力机制就像是聚光灯，它能够让模型聚焦于输入数据中的某些重要部分。在自然语言处理中，给定一个句子，注意力机制可以帮助模型关注到与当前任务最相关的单词。\n",
    "\n",
    "2. 多头注意力机制\n",
    "\n",
    "核心思想\n",
    "\n",
    "多头注意力机制是将一个大的注意力过程分解成多个小的注意力 “头”（head）并行地进行计算。每个头都可以学习到输入数据不同方面的特征。\n",
    "假设我们有一个输入序列 X ，多头注意力会将 X 分别经过多个不同的线性变换（权重矩阵不同），得到多个不同的表示$ Q_i、K_i、V_i$（i 表示第 i 个头）。然后每个头独立地进行注意力计算：\n",
    "Z = Attention($Q_i,K_i,V_i$) = Softmax($\\frac{Q_iK_i^T}{\\sqrt{d}}$)$V_i$\n",
    "\n",
    "，其中 $d_k$ 是键（K）向量的维度。\n",
    "最后，将各个头的结果进行拼接，并经过一个线性层得到最终的输出。\n",
    "3. 多头交叉注意力模块\n",
    "\n",
    "输入与输出\n",
    "\n",
    "在 transformers 架构中，多头交叉注意力模块通常用于解码器（Decoder）部分。它接收两个不同的输入：来自编码器（Encoder）的输出和解码器自身的输入。\n",
    "\n",
    "来自编码器的输出作为键（K）和值（V），解码器自身的输入作为查询（Q）。这就是 “交叉” 的含义，因为查询和键 / 值来自不同的来源。\n",
    "\n",
    "计算过程\n",
    "\n",
    "首先，对编码器的输出进行线性变换得到 K 和 V，对解码器的输入进行线性变换得到 Q。\n",
    "\n",
    "然后，和多头注意力机制类似，将 Q、K、V 分成多个头进行并行的注意力计算。\n",
    "\n",
    "例如，在机器翻译任务中，编码器处理源语言句子得到的表示作为  和 ，解码器在生成目标语言句子时的当前输入作为 。多头交叉注意力模块能够让解码器在生成每个单词时，根据编码器对源语言句子的编码结果（即  和 ）来动态地关注源语言句子中的不同部分。\n",
    "\n",
    "作用与意义\n",
    "\n",
    "多头交叉注意力模块有助于解码器更好地利用编码器的信息，从而生成更准确的输出。通过多个头的并行计算，它可以从不同角度捕捉编码器输出和解码器输入之间的关系，提高模型对输入序列的理解和生成能力。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2e62fea",
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderLayer(nn.Module):\n",
    "    def __init__(self,d_model,heads, dropout=0.1):\n",
    "        super().__init__()\n",
    "        self.norm_1 = Norm(d_model)\n",
    "        self.norm_2 = Norm(d_model)\n",
    "        self.attn = MultiHeadAttention(heads, d_model, dropout=dropout) #多头自注意力模块\n",
    "        self.ff = FeedForward(d_model, dropout=dropout)  #前馈网络\n",
    "        self.dropout_1 = nn.Dropout(dropout)\n",
    "        self.dropout_2 = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self,x,mask):\n",
    "        attn_output = self.attn(x,x,x,mask)\n",
    "        attn_output = self.dropout_1(attn_output)\n",
    "        x= x + attn_output\n",
    "        x= self.norm_1(x)\n",
    "        ff_output = self.ff(x)\n",
    "        ff_output= self.dropout_2(ff_output)\n",
    "        x=x + ff_output\n",
    "        x=self.norm_2(x)\n",
    "        return x\n",
    "    \n",
    "class Encoder(nn.Module):\n",
    "    def __init__(self,vocab_size, d_model, N, heads, dropout):\n",
    "        super().__init__()\n",
    "        self.N = N\n",
    "        self.embed = Embedder(vocab_size,d_model) #embeding\n",
    "        self.pe = PositionalEncoder(d_model,dropout=dropout) #位置编码\n",
    "        self.layers = get_clones(EncoderLayer(d_model, heads, dropout), N)  \n",
    "        self.norm = Norm(d_model) #归一化\n",
    "        \n",
    "    def forward(self, src, mask):\n",
    "        x = self.embed(src) #\n",
    "        x = self.pe(x)\n",
    "        for i in range(self.N):\n",
    "            x= self.layers[i](x, mask)\n",
    "        return self.norm(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d8f5371",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DecoderLayer(nn.Module):\n",
    "    def __init__(self,d_model, heads, dropout=0.1):\n",
    "        super().__init__()\n",
    "        self.norm_1 = Norm(d_model)\n",
    "        self.norm_2 = Norm(d_model)\n",
    "        self.norm_3 = Norm(d_model)\n",
    "        self.dropout_1 = nn.Dropout(dropout)\n",
    "        self.dropout_2 = nn.Dropout(dropout)\n",
    "        self.dropout_3 = nn.Dropout(dropout)\n",
    "        self.attn_1=MultiHeadAttention(heads, d_model, dropout=dropout)\n",
    "        self.attn_2=MultiHeadAttention(heads, d_model, dropout=dropout)\n",
    "        self.ff=FeedForvard(d_model,dropout=dropout)\n",
    "        \n",
    "    def forward(self,x,e_outputs,src_mask, trg_mask):\n",
    "        attn_output_1=self.attn_1(x,x,x,trg_mask)\n",
    "        attn_output_1=self.dropout_1(attn_output_1)\n",
    "        x= x + attn_output_1\n",
    "        x= self.norm_1(x)\n",
    "        attn_output_2 = self.attn_2(x,e_outputs, e_outputs, src_mask)\n",
    "        attn_output_2 = self.dropout_2(attn_output_2)\n",
    "        x = x + attn_output_2\n",
    "        x = self.norm_2(x)\n",
    "        \n",
    "        ff_output = self.ff(x)\n",
    "        ff_output = self.dropout_3(ff_output)\n",
    "        x = x+ ff_output\n",
    "        z = self.norm_3(x)\n",
    "        \n",
    "        return x\n",
    "    \n",
    "class Decoder(nn,Module):\n",
    "    def __init__(self,vocab_size,d_model,N, heads, dropout):\n",
    "        super().__init__()\n",
    "        self.N = N\n",
    "        self.embed = Embedder(vocab_size,d_model)\n",
    "        self.pe = PositionalEncoder(d_model,dropout=dropout)\n",
    "        self.layers = get_clones(DecoderLayer(d_model, heads, dropout), N)\n",
    "        self.norm = Norm(d_model)\n",
    "    def forward(self,trg,e_outputs, src_mask, trg_mask):\n",
    "        x = self.embed(trg)\n",
    "        x = self.pe(x)\n",
    "        for i in range(self.N):\n",
    "            x = self.layers[i](x,e_outputs, src_mask, trg_mask)\n",
    "        return self.norm(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1731e2d5",
   "metadata": {},
   "source": [
    "# GPT\n",
    "\n",
    "生成式预训练语言模型（Generative Pre-Training，GPT）是典型的生成式预训练语言模型之一。GPT 的模型结构如图2.3 所示，它是由多层Transformer组成的单向语言模型，主要分为输入层、编码层和输出层三部分。\n",
    "\n",
    "![alt text](../images/GPT模型结构.png)\n",
    "\n",
    "GPT 采用生成式预训练方法，单向意味着模型只能从左到右或从右到左对文本序列建模，所采用的Transformer 结构和解码策略保证了输入文本每个位置只能依赖过去时刻的信息。\n",
    "\n",
    "给定文本序列$w = w_1w_2,...,w_n$，GPT 首先在输入层中将其映射为稠密的向量:(根据解析，就是进行embeding了)\n",
    "\n",
    "$v_i = v_i^t + v_i^p$\n",
    "\n",
    "随后将embeding的结果送入模型编码层。编码层由L 个Transformer 模块组成，在自注意力机制的作用下，每一层的每个表示向量都会包含之前位置表示向量的信息\n",
    "\n",
    "$h^{(L)} = Transformer-Block^{(L)}(h^{(0)})$\n",
    "\n",
    "GPT 模型的输出层基于最后一层的表示$h^{(L)}$，预测每个位置上的条件概率,计算过程表示为：\n",
    "\n",
    "$P(w_i|w_1,...,w_{i-1}) = Softmax(W^eh_i^{(L)} + b^(out))$\n",
    "\n",
    "$W^e ∈ R^{|V|*d}$ 为词向量矩阵，|V|为词表大小\n",
    "\n",
    "单向语言模型按照阅读顺序输入文本序列w，用常规语言模型目标优化w 的最大似然估计，使之能根据输入历史序列对当前词做出准确的预测：\n",
    "\n",
    "$L^{PT}(w) = - \\sum_{i=1}^{n}log P(w_i|w_0...w_i-1,θ)$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90cea215",
   "metadata": {},
   "source": [
    "# 稠密向量\n",
    "\n",
    "稠密向量的作用：为了让模型能够更好地处理这些输入，GPT 首先要将这种离散的、稀疏的文本表示转换为稠密的向量。这个过程通常是通过词嵌入（word embedding）技术来实现的。词嵌入会将每个单词映射到一个固定维度的向量空间中。例如，每个单词可能被映射到一个维度为 768（这是 GPT 模型中常见的维度）的向量。这个向量空间中的向量是稠密的，意味着向量的每个维度都有具体的值，而且相邻的向量之间可以表示相似的语义关系。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30f6263a",
   "metadata": {},
   "source": [
    "# 对$L^{PT}(w)$的理解\n",
    "\n",
    "### 单向语言模型的输入方式\n",
    "\n",
    "阅读顺序输入文本序列：想象你在阅读一本书，是按照从左到右（对于大多数语言）的顺序来阅读文字的。单向语言模型也是如此，它会按照文本的自然顺序，一个词接一个词地接收文本序列。例如，对于句子 “我喜欢读书”，它会先接收 “我”，然后是 “喜欢”，接着是 “读书” 这样的顺序。\n",
    "\n",
    "### 常规语言模型目标与最大似然估计\n",
    "\n",
    "常规语言模型目标：语言模型的主要目标是预测文本序列出现的概率。对于给定的一个词汇序列，它想要知道这个序列在语言中出现的可能性有多大。比如，“我喜欢读书” 这个句子在日常语言使用中有一定的概率出现，语言模型就要去估计这个概率。\n",
    "\n",
    "最大似然估计：这是一种统计方法，用于估计模型的参数。在语言模型中，我们希望找到一组参数（例如，在神经网络语言模型中，这些参数可能是神经网络的权重），使得观察到的文本序列（也就是我们输入给语言模型的真实文本）出现的概率最大。以刚才的句子为例，通过最大似然估计，我们要调整模型的参数，使得 “我喜欢读书” 这个句子被模型预测出来的概率尽可能高。\n",
    "\n",
    "### 根据输入历史序列预测当前词\n",
    "\n",
    "利用历史信息：还是用 “我喜欢读书” 这个句子，当模型要预测 “读书” 这个词的时候，它会根据之前已经输入的 “我喜欢” 这些历史序列信息。就好像我们在猜一个人接下来要说什么，会根据他之前已经说过的话来推测。模型会从之前输入的词中学习到语言的模式、语法规则和语义信息等。比如，“我喜欢” 后面可能是一个动词短语，模型根据学习到的知识就可以更准确地预测出 “读书” 这个词。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8aa44b2c",
   "metadata": {},
   "source": [
    "# 有监督下游任务微调\n",
    "\n",
    "通过无监督语言模型预训练，使得GPT 模型具备了一定的通用语义表示能力。下游任务微调（Downstream Task Fine-tuning）的目的是在通用语义表示的基础上，根据下游任务的特性进行适配。\n",
    "\n",
    "先将文本序列x 输入GPT 模型，获得最后一层的最后一个词所对应的隐藏层输出$h^{(L)}_n$(L层的transformers-block) ，在此基础上，通过全连接层变换结合Softmax 函数，得到标签预测结果。\n",
    "\n",
    "$P(y|x_1....x_n) = Softmax(h^{(L)}W_y)$\n",
    "\n",
    "通过对整个标注数据集D 优化如下目标函数精调下游任务：\n",
    "\n",
    "$L^{FT}(D) = - \\sum_{(x,y)}logP(y|x_1...x_n)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab664600",
   "metadata": {},
   "source": [
    "# 解决灾难性遗忘（Catastrophic Forgetting）问题\n",
    "\n",
    "因通常采用混合预训练任务损失和下游微调损失的方法来缓解上述问题。在实际应用中，通常采用如下公式进行下游任务微调：\n",
    "\n",
    "$L = L^{FT}(D) + λL^{PT}(D)$ \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a4aa4a5",
   "metadata": {},
   "source": [
    "# 灾难性遗忘\n",
    "\n",
    "### 定义\n",
    "\n",
    "灾难性遗忘是指模型在学习新的知识或任务时，会完全或大部分忘记之前已经学到的知识。在大模型（如深度神经网络）中，这是一个比较突出的问题。\n",
    "例如，一个预训练好的语言模型，最初能够很好地处理多种文本分类任务，如新闻分类、小说体裁分类等。但当对它进行新的任务训练，比如情感分析训练后，它可能会在原来的新闻分类和小说体裁分类任务上表现得很差，好像忘记了之前学到的相关知识。\n",
    "\n",
    "### 产生原因\n",
    "\n",
    "参数更新方式：在神经网络中，模型的学习是通过更新参数来实现的。当学习新任务时，为了优化新任务的性能，模型会调整参数。这些参数的改变可能会覆盖或破坏之前任务中已经学习到的有效参数设置。\n",
    "\n",
    "以一个简单的神经网络为例，假设它有用于分类任务的权重矩阵。在学习新任务时，反向传播算法会根据新任务的误差来更新这个权重矩阵，这种更新可能会导致之前任务相关的权重信息丢失。\n",
    "\n",
    "数据分布差异：如果新数据和旧数据的分布差异较大，模型可能会更侧重于学习新数据的模式，而忽略旧数据的模式。例如，一个图像识别模型最初是用自然风景图片训练的，数据分布主要是山川、河流等自然元素。当加入大量的城市建筑图片进行新的训练时，模型可能会因为两种数据分布的不同，而在学习城市建筑图片特征的过程中，忘记自然风景图片的特征。\n",
    "\n",
    "### 影响\n",
    "\n",
    "性能下降：对之前任务的性能产生严重的负面影响，使得之前已经训练好的模型部分失去作用。在实际应用中，这可能会导致模型在需要综合利用之前知识的场景下表现不佳。\n",
    "\n",
    "效率降低：如果要避免灾难性遗忘，可能需要重新训练模型或者采取复杂的训练策略，这会增加计算资源的消耗和训练时间。\n",
    "\n",
    "### 解决方法\n",
    "\n",
    "经验回放（Experience Replay）：类似于记忆回放，将之前学习任务的部分数据保存下来，在学习新任务的过程中，让模型重新学习这些旧数据。例如，在强化学习领域，智能体可以定期回顾之前经历过的状态、动作和奖励，以巩固旧知识。\n",
    "\n",
    "多任务学习（Multi - task Learning）：同时对多个任务进行训练，而不是逐个训练。通过共享模型的某些部分，让模型在学习新任务的同时，也能保留旧任务的知识。例如，在自然语言处理中，同时进行语法检查和语义理解两个任务的训练，使模型在两个任务之间找到平衡，减少遗忘。\n",
    "\n",
    "弹性权重巩固（Elastic Weight Consolidation）：在更新参数时，对与旧任务相关的重要参数施加约束，使这些参数在学习新任务时不会被过度修改。通过计算每个参数对于旧任务的重要性，然后在更新时考虑这种重要性来实现。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1b19fdb",
   "metadata": {},
   "source": [
    "# 混合预训练任务损失和下游微调损失\n",
    "\n",
    "### 基本概念\n",
    "\n",
    "预训练任务损失：在预训练阶段，模型（如 Transformer - based 语言模型）会在大规模的无标签数据上进行训练。这个过程中会有一个损失函数来衡量模型预测结果和实际数据之间的差距。例如，在掩码语言模型（Masked Language Model，MLM）任务中，模型需要预测被掩码的单词，预训练任务损失就是根据预测单词和真实单词之间的差异来计算的。\n",
    "\n",
    "下游微调损失：当模型完成预训练后，会在特定的下游任务（如文本分类、机器翻译等）上进行微调。在这个阶段，同样有一个损失函数来衡量模型在下游任务中的表现。例如，在文本分类任务中，下游微调损失可以是交叉熵损失，用于衡量模型预测的类别概率和实际类别之间的差异。\n",
    "\n",
    "混合损失：就是将预训练任务损失和下游微调损失结合起来，一起用于模型的训练和优化。\n",
    "\n",
    "### 混合的原因\n",
    "\n",
    "知识保留与迁移：预训练阶段模型学到了丰富的语言知识和通用模式。通过混合损失，在下游微调时可以更好地保留这些知识。例如，预训练时学会的词法、句法知识可以帮助下游任务（如情感分析）更好地理解文本，防止过度拟合下游任务的少量数据而丢失预训练获得的通用知识。\n",
    "\n",
    "提高泛化能力：有助于模型在不同任务和数据分布之间找到平衡。单一的下游微调损失可能会使模型过度适应下游任务的特定数据，而忽略了预训练的一般性知识。混合损失可以让模型在适应下游任务的同时，也能利用预训练阶段的泛化能力，从而在新的数据和任务场景中表现更好。\n",
    "\n",
    "### 混合方式\n",
    "\n",
    "加权求和：最常见的方式是给预训练任务损失和下游微调损失分别赋予一个权重，然后将它们相加作为总的损失。例如，设预训练任务损失为$L_pt$，下游微调损失为$L_ft$，总损失$L=αL_pt+βL_ft$，其中α和β是权重系数。权重的选择可以根据任务的性质、数据的规模等因素来确定。\n",
    "\n",
    "动态调整权重：在训练过程中，根据训练阶段或模型性能动态地调整预训练任务损失和下游微调损失的权重。例如，在训练初期，可能会给预训练任务损失较大的权重，让模型更多地依赖预训练知识；随着训练的进行，逐渐增加下游微调损失的权重，使模型更好地适应下游任务。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd3a25aa",
   "metadata": {},
   "source": [
    "# LLama3\n",
    "\n",
    "LLaMA 采用的Transformer 结构和细节与Transformer 结构的不同之处为：\n",
    "\n",
    " * 前置层归一化（Pre-normalization）\n",
    " * RMSNorm 归一化函数（Normalizing Function）\n",
    " * SwiGLU激活函数\n",
    " * 旋转位置嵌入（Rotary Positional Embeddings，RoPE）\n",
    "\n",
    " ![alt text](../images/LLama3结构图.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7661899e",
   "metadata": {},
   "source": [
    "# 总结\n",
    "\n",
    "以上是大模型的总体思维，随着大模型发展，在整个模型的结构上进行了优化，归一化函数、激活函数、位置编码方式等进行了优化。包括对注意力机制进行的优化。主要两种方法为\n",
    "\n",
    "从近似注意力出发，旨在减少注意力计算和内存需求，提出了稀疏近似、低秩近似等方法。\n",
    "\n",
    "从计算加速设备本身的特性出发，研究如何更好地利用硬件特性对Transformer 中的注意力层进行高效计算。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f52edf6",
   "metadata": {},
   "source": [
    "# 稀疏注意力机制\n",
    "\n",
    "对一些训练好的Transformer 结构中的注意力矩阵进行分析时发现，其中很多是稀疏的，因此可以通过限制Query-Key 对的数量来降低计算复杂度。这类方法称为稀疏注意力（SparseAttention）机制。可以将稀疏化方法进一步分成基于位置的和基于内容信息的两类。\n",
    "\n",
    "基于位置的稀疏注意力机制的基本类型如图2.6 所示，主要包含如下五种类型：全局注意力（Global Attention）、带状注意力（Band Attention）、膨胀注意力（Dilated Attention）、随机注意力（Random Attention）、局部块注意力（Block Local Attention）。\n",
    "\n",
    "![alt text](../images/稀疏注意力机制.png)\n",
    "\n",
    "现有的稀疏注意力机制，通常是基于上述五种基于位置的稀疏注意力机制的复合模式"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffb1cdb2",
   "metadata": {},
   "source": [
    "# FlashAttention\n",
    "\n",
    "FlashAttention就是通过利用GPU硬件中的特殊设计，针对全局内存和共享存储的I/O速度的不同，尽可能地避免HBM中读取或写入注意力矩阵。\n",
    "\n",
    "FlashAttention目标是尽可能高效地使用SRAM来加快计算速度，避免从全局内存中读取和写入注意力矩阵。达成该目标需要能做到在不访问整个输入的情况下计算Softmax函数，并且后向传播中不能存储中间注意力矩阵。\n",
    "\n",
    "FlashAttention 就提出了不使用中间注意力矩阵，通过存储归一化因子来减少全局内存消耗的方法。\n",
    "\n",
    "FlashAttention 算法并没有将S、P 整体写入全局内存，而是通过分块写入，存储前向传递的Softmax 归一化因子，在后向传播中快速重新计算片上注意力，这比从全局内存中读取中间注意力矩阵的标准方法更快。\n",
    "\n",
    "虽然大幅减少了全局内存的访问量，重新计算也导致FLOP 增加，但其运行的速度更快且使用的内存更少\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cbd864e",
   "metadata": {},
   "source": [
    "# 多查询注意力\n",
    "\n",
    "多查询注意力（Multi Query Attention）是多头注意力的一种变体。其特点是，在多查询注意力中不同的注意力头共享一个键和值的集合，每个头只单独保留了一份查询参数，因此键和值的矩阵仅有一份，这大幅减少了显存占用，使其更高效。\n",
    "由于多查询注意力改变了注意力机制的结构，因此模型通常需要从训练开始就支持多查询注意力。文献的研究结果表明，可以通过对已经训练好的模型进行微调来添加多查询注意力支持，仅需要约5% 的原始训练数据量就可以达到不错的效果。\n",
    "包括Falcon、SantaCoder、StarCoder 在内的很多模型都采用了多查询注意力机制。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "320cf5a2",
   "metadata": {},
   "source": [
    "# 现状\n",
    "\n",
    "归一化函数和激活函数的选择对于大语言模型的收敛性具有一定影响，因此在LLaMA 模型被提出之后，大多数开源模型沿用了RMSNorm（优化的归一化方式） 和SwiGLU（优化的激活函数） 的组合方式。\n",
    "\n",
    "由于LLaMA 模型所采用的位置编码方法RoPE（旋转位置嵌入）的外推能力不好，因此后续一些研究采用了ALiBi 等具有更好外推能力的位置编码方法，使模型具有更长的上下文建模能力。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f684ce64",
   "metadata": {},
   "source": [
    "# ALiBi\n",
    "\n",
    "### ALiBi 简介\n",
    "\n",
    "ALiBi 是 “Attention with Linear Biases”（带有线性偏差的注意力）的缩写，它是一种用于 Transformer 架构中的位置编码替代方案。在自然语言处理任务中，Transformer 架构通常需要某种方式来表示单词在句子中的位置信息，传统的方法如正弦 - 余弦位置编码等。ALiBi 提供了一种新的思路来融入位置信息。\n",
    "\n",
    "### ALiBi 的原理\n",
    "位置偏差的引入：ALiBi 通过在注意力机制中添加线性偏差来表示位置信息。具体来说，在计算注意力分数时，它会根据查询（Query）和键（Key）向量对应的位置来添加一个偏差项。这个偏差项是基于位置之间的相对距离线性计算得到的。\n",
    "\n",
    "数学表示：假设在多头注意力机制中有头的索引为，查询位置为，键位置为，那么带有 ALiBi 偏差的注意力分数计算公式为，其中和分别是位置和处的查询和键向量，是键向量的维度，是基于位置差的线性偏差。\n",
    "\n",
    "偏差计算示例：偏差通常是一个预先定义好的基于位置差的序列。例如，对于位置差为 1 的情况，偏差可能是一个较小的值；对于位置差为 5 的情况，偏差会是一个较大的值。这种偏差的大小是根据位置差线性变化的，从而实现了位置信息的融入。\n",
    "\n",
    "### ALiBi 的优势\n",
    "外推性能好：在处理长序列文本时，ALiBi 的位置编码方式具有更好的外推性能。相比传统的位置编码，它不需要对模型进行额外的调整就可以更好地处理超出训练时序列长度的文本。例如，在预训练模型时使用了一定长度的文本序列，当遇到更长的文本时，ALiBi 能够更自然地适应，而不会像一些传统位置编码方法那样出现性能急剧下降的情况。\n",
    "\n",
    "简单有效：它的实现相对简单，不需要复杂的三角函数计算（如正弦 - 余弦位置编码）。通过简单的线性偏差就可以有效地将位置信息融入到注意力机制中，并且在很多自然语言处理任务中取得了很好的效果，如文本生成、机器翻译等。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "p380",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
